| Challenge: | Comparative learning is a key component in fine-tuning code search models . however, negative samples of InfoNCE may deteriorate its representation learning . |
| Approach: | They propose a loss function that inserts weight terms into InfoNCE to improve contrastive learning. |
| Outcome: | The proposed loss function is a special case of Soft-InfoNCE, the authors show . it is more accurate than other loss functions, and it is faster than other models. |
Similar Papers
Improving Cross-lingual Transfer with Contrastive Negative Learning and Self-training (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent studies improve cross-lingual transfer learning by better aligning the internal representations within the multilingual model or exploring the information of the target language using self-training. |
| Approach: | They propose to use negative pairs to align the multilingual model and self-train the model to converge on the obtained clean pseudo-labels. |
| Outcome: | The proposed method improves upon the baseline models and can serve as a beneficial complement to the alignment-based methods. |
Debiased Contrastive Learning of Unsupervised Sentence Representations (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that contrastive learning improves pre-trained language models to derive high-quality sentence representations. |
| Approach: | They propose a framework to punish false negatives and generate noise-based negatives to guarantee the uniformity of the representation space. |
| Outcome: | The proposed framework improves pre-trained language models while pushing apart irrelevant negatives to guarantee the uniformity of the representation space. |
Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification (2021.emnlp-main)
Copied to clipboard
| Challenge: | Fine-grained classification tasks involve distinguishing between classes with subtle differences between them. |
| Approach: | They analyse fine-grained text classification tasks by embedding class relationships into a contrastive objective function to help differently weigh the positives and negatives. |
| Outcome: | The proposed model outperforms previous contrastive methods on emotion classification and sentiment analysis. |
Understanding Hard Negatives in Noise Contrastive Estimation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing theoretical results in contrastive learning focus on unconditional negative distributions. |
| Approach: | They propose a method where highest-scoring incorrect labels are chosen as negatives . they also propose 'hard negative mining' where negatives are used as negative examples . |
| Outcome: | The proposed approach achieves strong results on the task of zero-shot entity linking. |
ContrastiveMix: Overcoming Code-Mixing Dilemma in Cross-Lingual Transfer for Information Retrieval (2024.naacl-short)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have been widely adopted in cross-lingual transfer . however, training mPLMs on code-mixed data is counterproductive . |
| Approach: | They propose a method that disentangles contrastive loss between conflicting objectives . they propose mPLMs that train on code-mixed data for information retrieval . |
| Outcome: | The proposed model outperforms mDPR in cross-lingual transfer tasks . it is effective in sentence-level and token-level zero-shot tasks despite lack of target language data . |
Negative Sampling Techniques in Dense Retrieval: A Survey (2026.findings-eacl)
Copied to clipboard
| Challenge: | Information Retrieval (IR) is fundamental to many modern NLP applications. |
| Approach: | They propose a taxonomy that categorizes negative sampling techniques in dense IR . they analyze them with respect to trade-offs between effectiveness, computational cost, implementation difficulty . |
| Outcome: | The proposed taxonomy categorizes techniques using random, static/dynamically mined, and synthetic datasets. |
Label-aware Hard Negative Sampling Strategies with Momentum Contrastive Learning for Implicit Hate Speech Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for implicit hate speech detection do not have significant advantage over cross-entropy loss-based learning. |
| Approach: | They propose a label-aware hard negative sampling strategy that encourages the model to learn detailed features from hard negative samples instead of random batch. |
| Outcome: | The proposed models outperform existing models for implicit hate speech detection both in- and cross-datasets. |
Improving Contrastive Learning of Sentence Embeddings with Focal InfoNCE (2023.findings-emnlp)
Copied to clipboard
| Challenge: | SimCSE does not fully exploit the potential of hard negative samples in contrastive learning. |
| Approach: | They propose an unsupervised contrastive learning framework that combines SimCSE with hard negative mining to enhance the quality of sentence embeddings. |
| Outcome: | The proposed framework improves sentence embeddings on various STS benchmarks in terms of Spearman’s correlation, representation alignment and uniformity. |
Weighted Contrastive Learning With False Negative Control to Help Long-tailed Product Classification (2023.acl-industry)
Copied to clipboard
| Challenge: | Item categorization (IC) aims to classify a product into leaf nodes in a categorical taxonomy due to scarce supervision. |
| Approach: | They propose to use K-positive contrastive loss (KCL) to address IC task’s long-tail issue by re-weighting positive pairs in the KCL loss with a regularization that the sum of weights should be constrained to K+1 as close as possible. |
| Outcome: | The proposed method improves on the long-tail issue in the image classification task and when using text-based contrastive learning, it can be applied on the IC task. |
Smoothed Contrastive Learning for Unsupervised Sentence Embedding (2022.coling-1)
Copied to clipboard
| Challenge: | Unsupervised contrastive sentence embedding models use InfoNCE loss function . increasing batch size leads to performance degradation when it exceeds threshold . |
| Approach: | They propose a simple smoothing strategy upon the InfoNCE loss function to reduce the number of false-negative pairs in a batch without increasing the batch size. |
| Outcome: | The proposed smoothing strategy improves unsupervised SimCSE on semantic similarity tasks. |