Batch-Softmax Contrastive Loss for Pairwise Sentence Scoring Tasks (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent advances in machine learning have led to the use of contrastive loss for representation learning. |
| Approach: | They propose to use batch-softmax contrastive loss to train pairwise sentence embeddings . they propose to take a batch-softermax contrastitive loss and train it with different loss functions . |
| Outcome: | The proposed model improves on a number of datasets and pairwise sentence scoring tasks. |
Similar Papers
Smoothed Contrastive Learning for Unsupervised Sentence Embedding (2022.coling-1)
Copied to clipboard
| Challenge: | Unsupervised contrastive sentence embedding models use InfoNCE loss function . increasing batch size leads to performance degradation when it exceeds threshold . |
| Approach: | They propose a simple smoothing strategy upon the InfoNCE loss function to reduce the number of false-negative pairs in a batch without increasing the batch size. |
| Outcome: | The proposed smoothing strategy improves unsupervised SimCSE on semantic similarity tasks. |
Contrastive Data and Learning for Natural Language Processing (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Current NLP models heavily rely on effective representation learning algorithms. |
| Approach: | This tutorial introduces contrastive learning and provides an introduction to the techniques. |
| Outcome: | This tutorial provides an introduction to the fundamentals of contrastive learning approaches and the theory behind them. |
Learning with Contrastive Examples for Data-to-Text Generation (2020.coling-main)
Copied to clipboard
Yui Uehara, Tatsuya Ishigaki, Kasumi Aoki, Hiroshi Noji, Keiichi Goshima, Ichiro Kobayashi, Hiroya Takamura, Yusuke Miyao
| Challenge: | Existing models for data-to-text generation generate fluent but sometimes incorrect sentences . Existing studies show that using contrastive examples improves the ability of generating sentences with better lexical choice without degrading the fluency. |
| Approach: | They propose to use models trained on incorrect sentences and learning methods that exploit contrastive examples to reduce such errors. |
| Outcome: | The proposed models generate fluent sentences but often have problematic ones in terms of correctness. |
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for learning universal sentence embeddings are based on unsupervised approaches with only dropout as noise. |
| Approach: | They propose an unsupervised approach that takes an input sentence and predicts itself in a contrastive objective with only standard dropout used as noise. |
| Outcome: | The proposed framework performs on par with previous supervised approaches and can produce superior sentence embeddings from unlabeled or labeled data. |
Differentiable Data Augmentation for Contrastive Sentence Representation Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | a contrastive learning framework is used to fine-tune pre-trained language models with unlabeled sentences or labeled sentences. |
| Approach: | They propose a method that makes hard positives from unlabeled sentences . they use a prefix attached to a model to allow for differentiable data augmentation . |
| Outcome: | The proposed method yields significant improvements over existing methods under semi-supervised and supervised settings. |
Alleviating Over-smoothing for Unsupervised Sentence Representation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to learn better unsupervised sentence representations have been successful . over-smoothing problem in unsupervised sentences reduces the capacity of powerful PLMs . |
| Approach: | They propose a method to solve the over-smoothing problem in unsupervised sentence representations by combining negatives from PLMs intermediate layers. |
| Outcome: | The proposed method improves on different strong baselines on Semantic Textual Similarity and Transfer datasets. |
Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing contrastive methods for learning universal sentence embeddings have limitations due to their over-parameterization and poor performance under domain shift settings. |
| Approach: | They propose to integrate an Energy-based Hinge loss to enhance the pairwise discriminative power of contrastive learning for sentence embeddings by combining PLMs with energy-based learning. |
| Outcome: | Empirical results show that the proposed method improves on seven standard semantic textual similarity tasks and a domain-shifted STS task. |
Debiased Contrastive Learning of Unsupervised Sentence Representations (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that contrastive learning improves pre-trained language models to derive high-quality sentence representations. |
| Approach: | They propose a framework to punish false negatives and generate noise-based negatives to guarantee the uniformity of the representation space. |
| Outcome: | The proposed framework improves pre-trained language models while pushing apart irrelevant negatives to guarantee the uniformity of the representation space. |
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations (2021.acl-long)
Copied to clipboard
| Challenge: | Sentence embeddings are an important component of many natural language processing systems. |
| Approach: | They propose a self-supervised objective for learning universal sentence embeddings that does not require labelled training data. |
| Outcome: | The proposed approach closes the performance gap between unsupervised and supervised pretraining for universal sentence encoders. |
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval (2023.acl-long)
Copied to clipboard
| Challenge: | Contrastive learning is the dominant paradigm for learning text representations from parallel text, but finding negative examples can be expensive in terms of compute or manual effort. |
| Approach: | They propose a generative model for learning multilingual text embeddings which encourages source separation in multilingual contexts by an approximation. |
| Outcome: | The proposed model outperforms both a strong contrastive and generative baseline on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval. |