Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning (P19-1)
Copied to clipboard
| Challenge: | Encoder-decoder models for unsupervised sentence representation learning discard decoder after training . decoded sentences are often used to make better predictions of words in a given sentence . |
| Approach: | They propose two types of decoding functions whose inverse can be easily derived without expensive inverse calculation. |
| Outcome: | The proposed models can learn good representations from encoders and decoders without expensive calculations. |
Similar Papers
Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation (P18-1)
Copied to clipboard
| Challenge: | Existing encoder-decoder dialog models cannot output interpretable actions as in traditional systems. |
| Approach: | They propose an unsupervised discrete sentence representation learning method that integrates with existing encoder-decoder dialog models for interpretable response generation. |
| Outcome: | The proposed model can be integrated with existing encoder-decoder dialog models and discover interpretable semantics via either auto encoding or context predicting. |
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (N18-1)
Copied to clipboard
| Challenge: | Currently, unsupervised word embeddings are routinely trained on large amounts of raw text data. |
| Approach: | They propose to use unsupervised word embeddings to train distributed representations of sentences. |
| Outcome: | The proposed method outperforms state-of-the-art models on most benchmark tasks and is robust to the produced general-purpose sentence embeddings. |
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Sentence embeddings are useful for language processing tasks, but it is unclear how to produce them from encoder-decoder models. |
| Approach: | They investigate the effects of scaling up sentence encoders to 11B parameters on sentence embeddings from text-to-text transformers (T5) . |
| Outcome: | The proposed models outperform the previous best models on both SentEval and SentGLUE transfer tasks. |
A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the CHATGPT Era and Beyond (2024.eacl-long)
Copied to clipboard
Abhinav Ramesh Kashyap, Thanh-Tung Nguyen, Viktor Schlegel, Stefan Winkler, See-Kiong Ng, Soujanya Poria
| Challenge: | Sentence representations are a critical component in NLP applications such as retrieval, question answering, and text classification. |
| Approach: | They present a systematic review of the literature on sentence representations focusing mostly on deep learning models. |
| Outcome: | The proposed methods highlight the key contributions and challenges in this area and suggest potential avenues for improving the quality and efficiency of sentence representations. |
Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence (2023.findings-acl)
Copied to clipboard
| Challenge: | Sentence-level representations are beneficial for various natural language processing tasks. |
| Approach: | They propose a generative embedding inversion attack that reconstructs input sequences based only on their sentence embeddeds. |
| Outcome: | The proposed model outperforms previous embedding inversion attacks in classification metrics and generates coherent and contextually similar sentences as original inputs. |
TSDAE: Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding Learning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to learn sentence embeddings require labeled data, but it is expensive. |
| Approach: | They propose an unsupervised method which learns sentence embeddings using unlabeled data . they propose a transformer-based sequence denoising auto-encoder which can be used for training . |
| Outcome: | The proposed method outperforms existing methods on four datasets from heterogeneous domains. |
Learning Visually Grounded Sentence Representations (N18-1)
Copied to clipboard
| Challenge: | Unsupervised sentence representation models suffer from the grounding problem because of lack of association between symbols and external information. |
| Approach: | They train a sentence encoder to predict image features of a caption and use them as sentence representations. |
| Outcome: | The proposed model improves on word embeddings and word representations on standard benchmarks. |
Text Generation with Exemplar-based Adaptive Decoding (N19-1)
Copied to clipboard
| Challenge: | Empirical results show that the proposed model achieves strong performance and outperforms comparable baselines. |
| Approach: | They propose a conditioned text generation model that uses a template-based approach to generate content from input text. |
| Outcome: | The proposed model outperforms baselines on abstractive text summarization and data-to-text generation. |
Sentence Bottleneck Autoencoders from Transformer Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for pretraining a language model on text have been used for building models in NLP, but they do not work for sentence representations derived from pretrainer models based on tokens or basic pooling operations. |
| Approach: | They propose to build a sentence-level autoencoder from a pretrained transformer language model. |
| Outcome: | The proposed model achieves better quality than previous methods on text similarity and style transfer tasks while using fewer parameters than large pretrained models. |
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on contrastive learning for sentence embeddings are weak . researchers have started to use contrastive training to learn better unsupervised sentences. |
| Approach: | They propose an information-aggregated contrastive learning framework for learning unsupervised sentence embeddings. |
| Outcome: | The proposed framework outperforms SimCSE on several benchmark datasets w.r.t the semantic text similarity task. |