Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency (D18-1)
Copied to clipboard
| Challenge: | Conditional models are frequently encountered in practice, but there has not been a rigorous theoretical analysis of NCE in this setting. |
| Approach: | They propose to use a ranking-based and ranking-only method for conditional models to estimate parameter estimates. |
| Outcome: | The proposed method avoids calculation of partition function or derivatives at each training step . it is closely related to negative sampling methods, now widely used in NLP . |
Similar Papers
Learning with Noise-Contrastive Estimation: Easing training by learning to scale (C18-1)
Copied to clipboard
| Challenge: | Neural language models have recently shown great improvement, but they share a common issue: large output vocabulary, computational time, and high dimensional space. |
| Approach: | They propose to make scaling factor a trainable parameter and use noise distribution to initialize output bias. |
| Outcome: | The proposed training strategies yield stable and competitive performances in small and large scale language modelling tasks. |
Adversarial Contrastive Estimation (P18-1)
Copied to clipboard
| Challenge: | Noise contrastive estimation (NCE) is a general strategy used in word embeddings and translations for knowledge graphs. |
| Approach: | They propose to augment negative sampler into mixture distribution with adversarially learned sampler and to combine it with noise contrastive estimation (NCE) they observe faster convergence and improved results on multiple metrics. |
| Outcome: | The proposed model performs better on word embeddings, order embedds and knowledge graph embeddments and faster convergence and improved results on multiple metrics. |
Self-Normalization Properties of Language Modeling (C18-1)
Copied to clipboard
| Challenge: | Existing methods to reduce run-times for language models with large word vocabularies are based on noise contrastive estimation (NCE) |
| Approach: | They propose to use noise-constrained noise-based models to approximate the normalized probability of a class without having to compute the partition function. |
| Outcome: | The proposed model outperforms softmax-based models in a variety of NLP tasks and is based on the noise-constrained noise-constant estimation properties. |
Understanding Hard Negatives in Noise Contrastive Estimation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing theoretical results in contrastive learning focus on unconditional negative distributions. |
| Approach: | They propose a method where highest-scoring incorrect labels are chosen as negatives . they also propose 'hard negative mining' where negatives are used as negative examples . |
| Outcome: | The proposed approach achieves strong results on the task of zero-shot entity linking. |
Conditional Semantic Textual Similarity via Conditional Contrastive Learning (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to assess similarity between sentences encounter over-estimation problem . compared to fuzzy representations, similarity is comparatively lower in terms of "The person's age". |
| Approach: | They propose a conditional contrastive learning framework that constructs positive and negative samples from two perspectives. |
| Outcome: | The proposed method achieves state-of-the-art performance with five models based on bi-encoder and tri-encoding architectures. |
Unified Interpretation of Softmax Cross-Entropy and Negative Sampling: With Case Study for Knowledge Graph Embedding (2021.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge graph embedding methods do not provide a fairly accurate comparison of the two loss functions. |
| Approach: | They propose to use the Bregman divergence to provide a unified interpretation of the softmax cross-entropy and negative sampling loss functions. |
| Outcome: | The proposed model can be used to predict missing relational links between entities using a scoring method. |
Con-NAT: Contrastive Non-autoregressive Neural Machine Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Neural machine translation models are autoregressive, which means they predict tokens one by one based on source tokens and previously predicted tokens. |
| Approach: | They propose a conditional masked language model which incorporates contrastive learning into the conditional language model. |
| Outcome: | The proposed model improves on WMT’16 Ro-En translation directions with different data sizes. |
The Trade-offs of Domain Adaptation for Neural Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Neural Language Models (LMs) trained on large generic training sets have been shown to be effective at adapting to smaller, specific target domains for language modeling and other downstream tasks. |
| Approach: | They propose a framework for a Neural Language Models (LM) to be presented in a common framework. |
| Outcome: | The proposed framework highlights similarities and subtle differences between adaptation techniques and the framework. |
Sampling Matters! An Empirical Study of Negative Sampling Strategies for Learning of Matching Models in Retrieval-based Dialogue Systems (D19-1)
Copied to clipboard
| Challenge: | Existing studies focus on constructing a matching model with sophisticated neural architectures, but do little to how to effectively learn such architectures from data. |
| Approach: | They propose to sample negative examples to automatically construct a training set for effective model learning in retrieval-based dialogue systems by using four sampling strategies. |
| Outcome: | The proposed learning method improves the performance of matching models on two benchmarks with three matching models. |
Noise-Robust Training with Dynamic Loss and Contrastive Learning for Distantly-Supervised Named Entity Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a task in natural language processing that aims at locating entity mentions in a given sentence and assigning them to certain types. |
| Approach: | They propose to use a dynamic loss function to better adapt to the changing noise during the training process and incorporate token level contrastive learning to fully utilize the noisy data. |
| Outcome: | The proposed method outperforms existing NER models on three benchmark datasets and outperformed existing models by significant margins. |