| Challenge: | Existing methods to reduce run-times for language models with large word vocabularies are based on noise contrastive estimation (NCE) |
| Approach: | They propose to use noise-constrained noise-based models to approximate the normalized probability of a class without having to compute the partition function. |
| Outcome: | The proposed model outperforms softmax-based models in a variety of NLP tasks and is based on the noise-constrained noise-constant estimation properties. |
Similar Papers
Learning with Noise-Contrastive Estimation: Easing training by learning to scale (C18-1)
Copied to clipboard
| Challenge: | Neural language models have recently shown great improvement, but they share a common issue: large output vocabulary, computational time, and high dimensional space. |
| Approach: | They propose to make scaling factor a trainable parameter and use noise distribution to initialize output bias. |
| Outcome: | The proposed training strategies yield stable and competitive performances in small and large scale language modelling tasks. |
Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency (D18-1)
Copied to clipboard
| Challenge: | Conditional models are frequently encountered in practice, but there has not been a rigorous theoretical analysis of NCE in this setting. |
| Approach: | They propose to use a ranking-based and ranking-only method for conditional models to estimate parameter estimates. |
| Outcome: | The proposed method avoids calculation of partition function or derivatives at each training step . it is closely related to negative sampling methods, now widely used in NLP . |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
Do self-supervised speech models develop human-like perception biases? (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in speech recognition and representation learning show that self-supervised pretraining is an excellent way of improving performance while reducing the amount of labelled data needed for training. |
| Approach: | They compare the representational spaces of wav2vec, HuBERT and contrastive predictive coding (CPC) with the perceptual spaces of French-speaking and English-speaking human listeners. |
| Outcome: | The proposed models capture fine-grained perceptual phenomena while supervised models are better at capturing coarser, phone-level effects and effects of listeners’ native language on perception. |
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP (2021.tacl-1)
Copied to clipboard
| Challenge: | Pretrained language models pick up and reproduce undesirable biases when trained on large, unfiltered crawls from the Internet. |
| Approach: | They propose a decoding algorithm that, given only a textual description of the undesired behavior, reduces the probability of a language model producing problematic text. |
| Outcome: | The proposed approach reduces the probability of a language model producing problematic text by giving only a textual description of the undesired behavior. |
Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | In the recent past, a popular way of evaluating natural language understanding was to consider a model’s ability to perform natural language inference (NLI) tasks. |
| Approach: | They focus on five different NLI benchmarks across six models of different scales and examine how their accuracies develop during training. |
| Outcome: | The softmax distributions of models align with human label distributions in cases where statements are ambiguous or vague. |
Mind Your Bias: A Critical Review of Bias Detection Methods for Contextual Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for detection of biases in contextual language models are inconsistent and inconclusive. |
| Approach: | They propose to use word embedding association test to detect biases in contextual language models to compare them with other methods. |
| Outcome: | The proposed methods are inconsistent and inconclusive for language models with word embeddings. |
Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Xue et al., 2021) show that large language models suffer from performance degradation on unseen closely-related languages and dialects relative to their high-resource language neighbour (HRLN). |
| Approach: | They propose to model phonological, morphological, and lexical distance as Bayesian noise processes to synthesize artificial languages that are controllably distant from the HRLN. |
| Outcome: | The proposed model offers insights on model robustness to isolated and composed linguistic phenomena and the impact of task and HRL characteristics on PD. |
A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on syntactic knowledge models has not provided a clear picture of the properties required to produce proper syntaktic generalizations. |
| Approach: | They propose to evaluate syntactic knowledge of language models by varying model architectures . they find substantial differences in syntaktic generalization performance by model architecture . |
| Outcome: | The proposed model architectures outperform other architectures on a set of 34 English-language syntactic test suites. |