Smoothed Contrastive Learning for Unsupervised Sentence Embedding (2022.coling-1)
Copied to clipboard
| Challenge: | Unsupervised contrastive sentence embedding models use InfoNCE loss function . increasing batch size leads to performance degradation when it exceeds threshold . |
| Approach: | They propose a simple smoothing strategy upon the InfoNCE loss function to reduce the number of false-negative pairs in a batch without increasing the batch size. |
| Outcome: | The proposed smoothing strategy improves unsupervised SimCSE on semantic similarity tasks. |
Similar Papers
Improving Contrastive Learning of Sentence Embeddings with Focal InfoNCE (2023.findings-emnlp)
Copied to clipboard
| Challenge: | SimCSE does not fully exploit the potential of hard negative samples in contrastive learning. |
| Approach: | They propose an unsupervised contrastive learning framework that combines SimCSE with hard negative mining to enhance the quality of sentence embeddings. |
| Outcome: | The proposed framework improves sentence embeddings on various STS benchmarks in terms of Spearman’s correlation, representation alignment and uniformity. |
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for learning universal sentence embeddings are based on unsupervised approaches with only dropout as noise. |
| Approach: | They propose an unsupervised approach that takes an input sentence and predicts itself in a contrastive objective with only standard dropout used as noise. |
| Outcome: | The proposed framework performs on par with previous supervised approaches and can produce superior sentence embeddings from unlabeled or labeled data. |
Alleviating Over-smoothing for Unsupervised Sentence Representation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to learn better unsupervised sentence representations have been successful . over-smoothing problem in unsupervised sentences reduces the capacity of powerful PLMs . |
| Approach: | They propose a method to solve the over-smoothing problem in unsupervised sentence representations by combining negatives from PLMs intermediate layers. |
| Outcome: | The proposed method improves on different strong baselines on Semantic Textual Similarity and Transfer datasets. |
ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding (2022.coling-1)
Copied to clipboard
| Challenge: | a new method for learning unsupervised sentence embeddings is proposed . unsup-SimCSE is biased because of the length information encoded into the sentence embeds . |
| Approach: | They propose a new unsupervised sentence embedding method that uses dropout to obtain positive pairs from a pre-trained Transformer encoder. |
| Outcome: | The proposed method outperforms the state-of-the-art unsup-SimCSE on a STS task. |
InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on contrastive learning for sentence embeddings are weak . researchers have started to use contrastive training to learn better unsupervised sentences. |
| Approach: | They propose an information-aggregated contrastive learning framework for learning unsupervised sentence embeddings. |
| Outcome: | The proposed framework outperforms SimCSE on several benchmark datasets w.r.t the semantic text similarity task. |
An Information Minimization Based Contrastive Learning Model for Unsupervised Sentence Embeddings Learning (2022.coling-1)
Copied to clipboard
| Challenge: | Recent contrastive learning methods keep positive pairs similar and push negative pairs apart, which leads to redundant information in sentence embeddings. |
| Approach: | They propose a contrastive learning approach which maximizes mutual information and minimizes the information entropy between positive and negative instances. |
| Outcome: | The proposed model outperforms all previous competitors on supervised and unsupervised tasks. |
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings (2022.naacl-main)
Copied to clipboard
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, James Glass
| Challenge: | Recent work shows that finetuning pretrained models with contrastive learning makes it possible to learn good sentence embeddings without labeled data. |
| Approach: | They propose an unsupervised contrastive learning framework for learning sentence embeddings . they use a masked language model to mask out the edited sentence . |
| Outcome: | The proposed framework outperforms SimCSE on semantic textual similarity tasks by 2.3 absolute points. |
Batch-Softmax Contrastive Loss for Pairwise Sentence Scoring Tasks (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent advances in machine learning have led to the use of contrastive loss for representation learning. |
| Approach: | They propose to use batch-softmax contrastive loss to train pairwise sentence embeddings . they propose to take a batch-softermax contrastitive loss and train it with different loss functions . |
| Outcome: | The proposed model improves on a number of datasets and pairwise sentence scoring tasks. |
SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that combining both proposed methods leads to a gain of 1.8 points compared to the strong baseline SimCSE configured with BERT base. |
| Approach: | They propose a method to deal with dropout noise and a dimension-wise contrastive learning objective to address feature corruption. |
| Outcome: | The proposed method achieves 1.8 points compared to the strong baseline SimCSE and 1.4 points for DiffCSE. |
More Discriminative Sentence Embeddings via Semantic Graph Smoothing (2024.eacl-short)
Copied to clipboard
| Challenge: | Text categorization is a natural language processing task that involves arranging texts into coherent groups based on their content. |
| Approach: | They propose to use semantic graph smoothing to enhance sentence embeddings from pretrained models to improve results for supervised and unsupervised document categorization tasks. |
| Outcome: | The proposed method improves sentences embeddings for supervised and unsupervised document categorization tasks. |