Learning with Noise-Contrastive Estimation: Easing training by learning to scale (C18-1)
Copied to clipboard
| Challenge: | Neural language models have recently shown great improvement, but they share a common issue: large output vocabulary, computational time, and high dimensional space. |
| Approach: | They propose to make scaling factor a trainable parameter and use noise distribution to initialize output bias. |
| Outcome: | The proposed training strategies yield stable and competitive performances in small and large scale language modelling tasks. |
Similar Papers
Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency (D18-1)
Copied to clipboard
| Challenge: | Conditional models are frequently encountered in practice, but there has not been a rigorous theoretical analysis of NCE in this setting. |
| Approach: | They propose to use a ranking-based and ranking-only method for conditional models to estimate parameter estimates. |
| Outcome: | The proposed method avoids calculation of partition function or derivatives at each training step . it is closely related to negative sampling methods, now widely used in NLP . |
Self-Normalization Properties of Language Modeling (C18-1)
Copied to clipboard
| Challenge: | Existing methods to reduce run-times for language models with large word vocabularies are based on noise contrastive estimation (NCE) |
| Approach: | They propose to use noise-constrained noise-based models to approximate the normalized probability of a class without having to compute the partition function. |
| Outcome: | The proposed model outperforms softmax-based models in a variety of NLP tasks and is based on the noise-constrained noise-constant estimation properties. |
Adversarial Contrastive Estimation (P18-1)
Copied to clipboard
| Challenge: | Noise contrastive estimation (NCE) is a general strategy used in word embeddings and translations for knowledge graphs. |
| Approach: | They propose to augment negative sampler into mixture distribution with adversarially learned sampler and to combine it with noise contrastive estimation (NCE) they observe faster convergence and improved results on multiple metrics. |
| Outcome: | The proposed model performs better on word embeddings, order embedds and knowledge graph embeddments and faster convergence and improved results on multiple metrics. |
Contrastive Learning for Task-Independent SpeechLLM-Pretraining (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models excel in speech processing tasks but their reliance on written text limits their application in real-world scenarios. |
| Approach: | They propose a task-independent speech pretraining stage and task-specific fine-tuning stage to adapt LLMs to speech processing tasks. |
| Outcome: | The proposed model outperforms models specialized on speech translation and question answering while being trained on 10% of the task-specific data. |
Noise-Robust Training with Dynamic Loss and Contrastive Learning for Distantly-Supervised Named Entity Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a task in natural language processing that aims at locating entity mentions in a given sentence and assigning them to certain types. |
| Approach: | They propose to use a dynamic loss function to better adapt to the changing noise during the training process and incorporate token level contrastive learning to fully utilize the noisy data. |
| Outcome: | The proposed method outperforms existing NER models on three benchmark datasets and outperformed existing models by significant margins. |
Contrastive Data and Learning for Natural Language Processing (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Current NLP models heavily rely on effective representation learning algorithms. |
| Approach: | This tutorial introduces contrastive learning and provides an introduction to the techniques. |
| Outcome: | This tutorial provides an introduction to the fundamentals of contrastive learning approaches and the theory behind them. |
The Trade-offs of Domain Adaptation for Neural Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Neural Language Models (LMs) trained on large generic training sets have been shown to be effective at adapting to smaller, specific target domains for language modeling and other downstream tasks. |
| Approach: | They propose a framework for a Neural Language Models (LM) to be presented in a common framework. |
| Outcome: | The proposed framework highlights similarities and subtle differences between adaptation techniques and the framework. |
Sequence-level Large Language Model Training with Contrastive Preference Optimization (2025.findings-naacl)
Copied to clipboard
| Challenge: | a new method to improve the performance of large language models requires a small computational cost. |
| Approach: | They propose a CPO procedure that can inject sequence-level information into the model at any training stage without expensive human labeled data. |
| Outcome: | The proposed objective surpasses the next token prediction in terms of win rate in instruction-following and text generation tasks. |
Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach (2021.naacl-main)
Copied to clipboard
| Challenge: | Fine-tuned pre-trained language models (LMs) have enormous success in many natural language processing tasks, but they still require excessive labeled data in the fine-tuning stage. |
| Approach: | They propose a framework to enable fine-tuning pre-trained language models with weak supervision without any labeled data. |
| Outcome: | The proposed framework outperforms the strongest baseline and achieves competitive performance with fully-supervised fine-tuning methods. |
Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding (2024.naacl-long)
Copied to clipboard
| Challenge: | End-to-end learning models require large volume of speech data with intent labels . however, models are sensitive to inconsistencies between training and evaluation conditions . |
| Approach: | They propose a module-based approach to learn intent in a noisy-channel model . they correlate error patterns between clean and noisy ASR transcripts . |
| Outcome: | The proposed method outperforms existing methods and improves in noisy environments. |