Role of Context in Unsupervised Sentence Representation Learning: the Case of Dialog Act Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Unsupervised learning of word representations involves capturing the contextual information surrounding word occurrences. |
| Approach: | They propose to use text-based dialog act tags to compare content- and context-oriented sentence representations inferred on telephone conversations to examine whether a contextual signal is of any significant benefit to general-purpose sentence representation. |
| Outcome: | The proposed model outperforms content-based and context-oriented representations on telephone conversations and shows that it increases the dimensionality of the vectors. |
Similar Papers
Contextualized Word Representations for Reading Comprehension (N18-2)
Copied to clipboard
| Challenge: | Reading comprehension (RC) is a high-level task in natural language understanding that requires reading a document and answering questions about its content. |
| Approach: | They propose to provide a standard neural network for reading a document and answering a question about its content. |
| Outcome: | The proposed model improves on the competitive SQuAD dataset by providing rich contextualized word representations and allowing it to choose between context-dependent and context-independent representations. |
A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the CHATGPT Era and Beyond (2024.eacl-long)
Copied to clipboard
Abhinav Ramesh Kashyap, Thanh-Tung Nguyen, Viktor Schlegel, Stefan Winkler, See-Kiong Ng, Soujanya Poria
| Challenge: | Sentence representations are a critical component in NLP applications such as retrieval, question answering, and text classification. |
| Approach: | They present a systematic review of the literature on sentence representations focusing mostly on deep learning models. |
| Outcome: | The proposed methods highlight the key contributions and challenges in this area and suggest potential avenues for improving the quality and efficiency of sentence representations. |
Dissecting Contextual Word Embeddings: Architecture and Representation (D18-1)
Copied to clipboard
| Challenge: | Existing work on learning contextual representations has used LSTM-based biLMs, but there is no reason to believe this is effective. |
| Approach: | They propose to use pre-trained bidirectional language models to learn contextual word embeddings for four NLP tasks and to use them to study the effects of architecture on endtask accuracy. |
| Outcome: | The proposed models outperform word embeddings for four NLP tasks and all learn representations that vary with network depth. |
Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation (P18-1)
Copied to clipboard
| Challenge: | Existing encoder-decoder dialog models cannot output interpretable actions as in traditional systems. |
| Approach: | They propose an unsupervised discrete sentence representation learning method that integrates with existing encoder-decoder dialog models for interpretable response generation. |
| Outcome: | The proposed model can be integrated with existing encoder-decoder dialog models and discover interpretable semantics via either auto encoding or context predicting. |
Contextualized and Generalized Sentence Representations by Contrastive Self-Supervised Learning: A Case Study on Discourse Relation Analysis (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to learn contextualized and generalized sentence representations are limited by the size of manually annotated data. |
| Approach: | They propose a method to learn contextualized and generalized sentence representations using contrastive self-supervised learning. |
| Outcome: | The proposed method outperforms baseline methods based on BERT, XLNet, and RoBERTa in English and Japanese and outperformed strong baseline methods. |
SLM: Learning a Discourse Language Representation with Sentence Unshuffling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent models for learning discourse language representations focus on bottom or top-level representations, but they do not capture intermediate-size structures in natural languages such as sentences and the relationships among them. |
| Approach: | They propose a new objective for learning a discourse language representation in a self-supervised manner by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. |
| Outcome: | The proposed model improves the original BERT model on downstream tasks by large margins. |
How Furiously Can Colorless Green Ideas Sleep? Sentence Acceptability in Context (2020.tacl-1)
Copied to clipboard
| Challenge: | a recent study shows that context affects our perception of sentence acceptability, but few studies investigate how it affects language models. |
| Approach: | They compare acceptability ratings of sentences judged in isolation with a relevant context and with an irrelevant context. |
| Outcome: | The proposed model achieves state-of-the-art for unsupervised acceptability prediction. |
What Helps Transformers Recognize Conversational Structure? Importance of Context, Punctuation, and Labels in Dialog Act Recognition (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing punctuation in the transcripts has a massive effect on the models’ performance, and specific label set specificity does not affect dialog act segmentation performance. |
| Approach: | They apply two pre-trained transformer models to a conversation transcript as a sequence of dialog acts and achieve strong results on Switchboard Dialog Act and Meeting Recorder Dialog Act corpora. |
| Outcome: | The proposed models achieve 8.4% and 14.2% error rates on the Switchboard Dialog Act and Meeting Recorder Dialog Act corpora. |
Decontextualization: Making Sentences Stand-Alone (2021.tacl-1)
Copied to clipboard
| Challenge: | Taking excerpts of text can be problematic, as key pieces may not be explicit in a local window. |
| Approach: | They define a problem of sentence decontextualization by rewriting a sentence to be interpretable out of context while preserving its meaning. |
| Outcome: | The proposed method can be used in question answering and document understanding tasks. |
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (N18-1)
Copied to clipboard
| Challenge: | Currently, unsupervised word embeddings are routinely trained on large amounts of raw text data. |
| Approach: | They propose to use unsupervised word embeddings to train distributed representations of sentences. |
| Outcome: | The proposed method outperforms state-of-the-art models on most benchmark tasks and is robust to the produced general-purpose sentence embeddings. |