Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, Qi Zhang
| Challenge: | Existing research shows that BERT and RoBERTa are poor in sentence embeddings due to static token embeddable bias and ineffective BERT layers. |
| Approach: | They propose a novel contrastive learning method for better sentence embeddings by using a template denoising technique. |
| Outcome: | The proposed method achieves 2.29 and 2.58 points of improvement compared to SimCSE and RoBERTa in the unsupervised setting. |
Similar Papers
Self-Guided Contrastive Learning for BERT Sentence Representations (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to derive sentence embeddings from pre-trained Transformers are unclear . a self-guided training method is used to fine-tune BERT in a supervised fashion . |
| Approach: | They propose a contrastive learning method that utilizes self-guidance to improve BERT sentence representations. |
| Outcome: | The proposed method is more effective than baselines on diverse sentence-related tasks and robust to domain shifts. |
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for learning universal sentence embeddings are based on unsupervised approaches with only dropout as noise. |
| Approach: | They propose an unsupervised approach that takes an input sentence and predicts itself in a contrastive objective with only standard dropout used as noise. |
| Outcome: | The proposed framework performs on par with previous supervised approaches and can produce superior sentence embeddings from unlabeled or labeled data. |
A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to contrastive learning are heavily affected by superficial features like sentence length and syntax. |
| Approach: | They propose a semantic-aware contrastive learning framework for sentence embeddings that explores the pseudo-token space representation of a sentence while eliminating the impact of superficial features such as sentence length and syntax. |
| Outcome: | The proposed framework outperforms the state-of-the-art on six standard semantic textual similarity tasks while maintaining an additional queue to store the representation of sentence embeddings. |
BERT Has More to Offer: BERT Layers Combination Yields Better Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Obtaining sentence representations from BERT-based models is valuable as it takes less time to pre-compute a one-time representation of the data and then use it for the downstream tasks. |
| Approach: | They propose to combine certain layers of a BERT-based model rested on the data set and model to achieve substantially better results. |
| Outcome: | The proposed method outperforms baseline models on seven semantic textual similarity datasets and on eight transfer data sets. |
Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration (2021.emnlp-main)
Copied to clipboard
| Challenge: | Phrase representations derived from pretrained language models often lack lexical similarity to determine semantic relatedness. |
| Approach: | They propose a contrastive fine-tuning objective that enables BERT to produce more powerful phrase embeddings by fine- tuning a dataset of diverse phrasal paraphrases and a large-scale dataset of phrases in context. |
| Outcome: | The proposed model outperforms baseline models across phrase-level similarity tasks while also showing increased lexical diversity between nearest neighbors in the vector space. |
Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing contrastive methods for learning universal sentence embeddings have limitations due to their over-parameterization and poor performance under domain shift settings. |
| Approach: | They propose to integrate an Energy-based Hinge loss to enhance the pairwise discriminative power of contrastive learning for sentence embeddings by combining PLMs with energy-based learning. |
| Outcome: | Empirical results show that the proposed method improves on seven standard semantic textual similarity tasks and a domain-shifted STS task. |
SKICSE: Sentence Knowable Information Prompted by LLMs Improves Contrastive Sentence Embeddings (2024.naacl-short)
Copied to clipboard
| Challenge: | Experimental results show that our model outperforms the previous state-of-the-art model GloVe on STS tasks. |
| Approach: | They propose a simple and effective prompt template that is able to obtain the knowable information of input sentences from LLMs. |
| Outcome: | The proposed model outperforms the previous state-of-the-art model PromptBERT on STS tasks. |
Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies focus on instance-wise contrastive learning, attempting to construct positive pairs with textual data augmentation. |
| Approach: | They propose a novel Contrastive learning method with Prompt-derived Virtual semantic prototypes that constructs virtual semantic prototype to each instance and derives negative prototypes by using the negative form of the prompts. |
| Outcome: | The proposed method performs on semantic textual similarity, transfer, and clustering tasks compared to baselines. |
Language-agnostic BERT Sentence Embedding (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning bilingual sentence embeddings are not well explored. |
| Approach: | They propose to combine best methods for learning multilingual sentence embeddings with pre-trained models to achieve 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba. |
| Outcome: | The proposed model achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, above the 65.5% achieved by LASER. |
On the Sentence Embeddings from Pre-trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained contextual representations like BERT have been widely used for NLP tasks. |
| Approach: | They propose to transform anisotropic sentence embedding distribution to smooth and isotropic Gaussian distribution by normalizing flows that are learned with an unsupervised objective. |
| Outcome: | The proposed method achieves significant performance gains over state-of-the-art embeddings on a variety of semantic textual similarity tasks. |