RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank (2023.acl-long)
Copied to clipboard
Jiduan Liu, Jiahao Liu, Qifan Wang, Jingang Wang, Wei Wu, Yunsen Xian, Dongyan Zhao, Kai Chen, Rui Yan
| Challenge: | Unsupervised sentence representation learning is one of the fundamental problems in natural language processing . contrastive learning methods fail to capture fine-grained ranking information among the sentences . |
| Approach: | They propose a novel approach for unsupervised sentence representation learning that integrates ranking consistency and ranking distillation with contrastive learning into a unified framework. |
| Outcome: | The proposed approach performs better over state-of-the-art models on STS and TR tasks. |
Similar Papers
Ranking-Enhanced Unsupervised Sentence Representation Learning (2023.acl-long)
Copied to clipboard
Yeon Seonwoo, Guoyin Wang, Changmin Seo, Sajal Choudhary, Jiwei Li, Xiang Li, Puyang Xu, Sunghyun Park, Alice Oh
| Challenge: | Unsupervised sentence representation learning has progressed through contrastive learning and data augmentation methods such as dropout masking. |
| Approach: | They propose a novel unsupervised sentence encoder, RankEncoder, which predicts the semantic vector of an input sentence by leveraging its relationship with other sentences in an external corpus. |
| Outcome: | The proposed unsupervised sentence encoder achieves 80.07% Spearman’s correlation, a 1.1% improvement over the previous state-of-the-art system. |
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for learning universal sentence embeddings are based on unsupervised approaches with only dropout as noise. |
| Approach: | They propose an unsupervised approach that takes an input sentence and predicts itself in a contrastive objective with only standard dropout used as noise. |
| Outcome: | The proposed framework performs on par with previous supervised approaches and can produce superior sentence embeddings from unlabeled or labeled data. |
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on contrastive learning for sentence embeddings are weak . researchers have started to use contrastive training to learn better unsupervised sentences. |
| Approach: | They propose an information-aggregated contrastive learning framework for learning unsupervised sentence embeddings. |
| Outcome: | The proposed framework outperforms SimCSE on several benchmark datasets w.r.t the semantic text similarity task. |
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings (2022.naacl-main)
Copied to clipboard
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, James Glass
| Challenge: | Recent work shows that finetuning pretrained models with contrastive learning makes it possible to learn good sentence embeddings without labeled data. |
| Approach: | They propose an unsupervised contrastive learning framework for learning sentence embeddings . they use a masked language model to mask out the edited sentence . |
| Outcome: | The proposed framework outperforms SimCSE on semantic textual similarity tasks by 2.3 absolute points. |
A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the CHATGPT Era and Beyond (2024.eacl-long)
Copied to clipboard
Abhinav Ramesh Kashyap, Thanh-Tung Nguyen, Viktor Schlegel, Stefan Winkler, See-Kiong Ng, Soujanya Poria
| Challenge: | Sentence representations are a critical component in NLP applications such as retrieval, question answering, and text classification. |
| Approach: | They present a systematic review of the literature on sentence representations focusing mostly on deep learning models. |
| Outcome: | The proposed methods highlight the key contributions and challenges in this area and suggest potential avenues for improving the quality and efficiency of sentence representations. |
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations (2021.acl-long)
Copied to clipboard
| Challenge: | Sentence embeddings are an important component of many natural language processing systems. |
| Approach: | They propose a self-supervised objective for learning universal sentence embeddings that does not require labelled training data. |
| Outcome: | The proposed approach closes the performance gap between unsupervised and supervised pretraining for universal sentence encoders. |
Alleviating Over-smoothing for Unsupervised Sentence Representation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to learn better unsupervised sentence representations have been successful . over-smoothing problem in unsupervised sentences reduces the capacity of powerful PLMs . |
| Approach: | They propose a method to solve the over-smoothing problem in unsupervised sentence representations by combining negatives from PLMs intermediate layers. |
| Outcome: | The proposed method improves on different strong baselines on Semantic Textual Similarity and Transfer datasets. |
Sentence Representations via Gaussian Embedding (2024.eacl-short)
Copied to clipboard
| Challenge: | Sentence embeddings represent a sentence's meaning as a point in a vector space and primarily use symmetric measures such as the cosine similarity to measure the similarity between sentences, they cannot capture asymmetric relationships between two sentences, such as entailment and hierarchical relations. |
| Approach: | They propose a Gaussian-distribution-based contrastive learning framework for sentence embedding that can handle asymmetric inter-sentential relations and a similarity measure for identifying entailment relations. |
| Outcome: | The proposed framework performs comparable to that of previous methods on natural language inference tasks and estimates direction of entailment relations, which is difficult with point representations. |
AugCSE: Contrastive Sentence Embedding with Diverse Augmentations (2022.aacl-main)
Copied to clipboard
| Challenge: | Similar work has shown that a single augmentation can be used to learn a robust generalpurpose representation with contrastive learning. |
| Approach: | They propose a unified framework to utilize diverse sets of data augmentations to achieve a better, general-purpose sentence embedding model. |
| Outcome: | The proposed framework achieves state-of-the-art results on downstream transfer tasks and performs competitively on semantic textual similarity tasks, using only unsupervised data. |
Composition-contrastive Learning for Sentence Embeddings (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work shows potential to learn vector representations from unlabelled data without task-specific fine-tuning. |
| Approach: | They propose to maximize alignment between textual embeddings and a composition of their phrasal constituents. |
| Outcome: | The proposed approach improves on similarity tasks comparable to state-of-the-art approaches. |