AugCSE: Contrastive Sentence Embedding with Diverse Augmentations (2022.aacl-main)
Copied to clipboard
| Challenge: | Similar work has shown that a single augmentation can be used to learn a robust generalpurpose representation with contrastive learning. |
| Approach: | They propose a unified framework to utilize diverse sets of data augmentations to achieve a better, general-purpose sentence embedding model. |
| Outcome: | The proposed framework achieves state-of-the-art results on downstream transfer tasks and performs competitively on semantic textual similarity tasks, using only unsupervised data. |
Similar Papers
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for learning universal sentence embeddings are based on unsupervised approaches with only dropout as noise. |
| Approach: | They propose an unsupervised approach that takes an input sentence and predicts itself in a contrastive objective with only standard dropout used as noise. |
| Outcome: | The proposed framework performs on par with previous supervised approaches and can produce superior sentence embeddings from unlabeled or labeled data. |
SDA: Simple Discrete Augmentation for Contrastive Sentence Representation Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for data augmentation have not been well explored. |
| Approach: | They propose to use punctuation insertion, modal verbs, and double negation to produce diverse forms of sentences. |
| Outcome: | The proposed methods perform better on diverse datasets with semantic similarity and standard negation. |
Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for data augmentation neglect fine-grained knowledge, such as entities and quantities, leading to insufficient diversity and high data noise. |
| Approach: | They propose a pipeline-based data augmentation method via LLMs and introduce the Gaussian-decayed gradient-assisted Contrastive Sentence Embedding (GCSE) model to enhance unsupervised sentence embeddings. |
| Outcome: | The proposed method achieves state-of-the-art performance in semantic textual similarity tasks using fewer data samples and smaller LLMs. |
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings (2022.naacl-main)
Copied to clipboard
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, James Glass
| Challenge: | Recent work shows that finetuning pretrained models with contrastive learning makes it possible to learn good sentence embeddings without labeled data. |
| Approach: | They propose an unsupervised contrastive learning framework for learning sentence embeddings . they use a masked language model to mask out the edited sentence . |
| Outcome: | The proposed framework outperforms SimCSE on semantic textual similarity tasks by 2.3 absolute points. |
Virtual Augmentation Supported Contrastive Learning of Sentence Representations (2022.findings-acl)
Copied to clipboard
| Challenge: | Despite profound successes, contrastive representation learning relies on carefully designed data augmentations using domain-specific knowledge. |
| Approach: | They propose a virtual augmentation supported Contrastive Learning of sentence representations . they approximate the neighborhood of an instance via its K-nearest in-batch neighbors . |
| Outcome: | The proposed model outperforms existing methods on a wide range of downstream tasks. |
ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding (2022.coling-1)
Copied to clipboard
| Challenge: | a new method for learning unsupervised sentence embeddings is proposed . unsup-SimCSE is biased because of the length information encoded into the sentence embeds . |
| Approach: | They propose a new unsupervised sentence embedding method that uses dropout to obtain positive pairs from a pre-trained Transformer encoder. |
| Outcome: | The proposed method outperforms the state-of-the-art unsup-SimCSE on a STS task. |
SumCSE: Summary as a transformation for Contrastive Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Sentence embedding models are typically trained using contrastive learning (CL) using human annotations directly or by repurposing other annotated datasets. |
| Approach: | They propose to use generative language models to generate CL data using annotated data. |
| Outcome: | The proposed method outperforms the previous best unsupervised method by 1.8 points and SimCSE, a strong supervised baseline by 0.3 points on the semantic text similarity (STS) benchmark. |
Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing contrastive methods for learning universal sentence embeddings have limitations due to their over-parameterization and poor performance under domain shift settings. |
| Approach: | They propose to integrate an Energy-based Hinge loss to enhance the pairwise discriminative power of contrastive learning for sentence embeddings by combining PLMs with energy-based learning. |
| Outcome: | Empirical results show that the proposed method improves on seven standard semantic textual similarity tasks and a domain-shifted STS task. |
PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learning sentence embeddings in unsupervised manner depend on mono-augmenting . existing approaches depend on augmenting biases and thus corrupt the quality of sentence embeds. |
| Approach: | They propose a method to augment a sentence with a semantically-close positive instance to construct contrastive pairs in unsupervised manner. |
| Outcome: | The proposed method improves performance on STS benchmarks and compares with existing methods. |
One Sentence, Two Embeddings: Contrastive Learning of Explicit and Implicit Semantic Representations (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing sentence embedding methods lack the ability to capture the implicit semantics of sentences. |
| Approach: | They propose a sentence embedding method that assigns two embeddables to each sentence . one represents the explicit semantics and the other represents the implicit semantics . results show DualCSE can effectively encode both explicit and implicit meanings - they argue . |
| Outcome: | The proposed method can effectively encode both explicit and implicit meanings and improve the performance of the downstream task. |