Stolen Probability: A Structural Weakness of Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Neural Network Language Models (NNLMs) generate probability distributions by using a softmax function. |
| Approach: | They propose to use a dot-product distance metric to generate probability distributions . they show that the dot product distance meter introduces a limitation bounding expressiveness . |
| Outcome: | The proposed model optimizes well with the inductive bias of NNLMs, but it results in suboptimal ordering of the embedding space. |
Similar Papers
Softmax Bottleneck Makes Language Models Unable to Represent Multi-mode Word Distributions (2022.acl-long)
Copied to clipboard
| Challenge: | Existing language models can't predict the probability distribution over next words or masked words because of a single hidden state. |
| Approach: | They propose a multi-facet softmax to address the limitations of softmax . they propose LMs that can predict the next word distribution better as their sizes grow . |
| Outcome: | The proposed model achieves two-fold improvements in the perplexity of GPT-2 and BERT. |
On Efficiently Representing Regular Languages as RNNs (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). |
| Approach: | They generalize their construction and show that RNNs can efficiently represent a larger class of LMs than previously claimed. |
| Outcome: | The results suggest that RNNs can represent a larger class of LMs than previously claimed . |
Information Locality as an Inductive Bias for Neural Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Inductive biases are inherent in every machine learning system, argues a new study . m-local entropy measures how well symbols disambiguate the next symbol . |
| Approach: | They propose a framework that captures local uncertainty of a language by quantifying how effectively preceding symbols disambiguate the next symbol. |
| Outcome: | The proposed framework captures the local uncertainty of a language by quantifying how effectively symbols disambiguate the next symbol. |
The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translation (2024.naacl-short)
Copied to clipboard
| Challenge: | Continuous-output neural machine translation models are trained to predict the continuous representation based on distances between vectors. |
| Approach: | They propose a continuous-output neural machine translation (CoNMT) approach that uses random output embeddings to outperform laboriously pre-trained models. |
| Outcome: | The proposed strategy outperforms pre-trained embeddings on large datasets and is strongest for rare words due to the geometry of their embedders. |
How to represent a word and predict it, too: Improving tied architectures for language modelling (D18-1)
Copied to clipboard
| Challenge: | Recent state-of-the-art models use word embeddings as input and output mappings instead of tied models. |
| Approach: | They propose to decouple hidden state from word embedding prediction . they extend their proposed modification to word2vec models . |
| Outcome: | The proposed architectures achieve comparable or better results compared to previous models without tying . the proposed architecture reduces parameters, enabling more compact models and faster learning. |
Quantity doesn’t buy quality syntax with neural language models (D19-1)
Copied to clipboard
| Challenge: | Recurrent neural network language models can learn to predict upcoming words with remarkably low perplexity . but in syntactically complex contexts, they often assign unexpectedly high probabilities to ungrammatical words . |
| Approach: | They investigate whether recurrent neural networks can learn to predict upcoming words with remarkably low perplexity. |
| Outcome: | The proposed models perform worse than GPT and BERT in some constructions than LSTMs in other contexts. |
Revisiting Simple Neural Probabilistic Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent advances in language modeling have been driven not only by advances in neural architectures, but also through hardware and optimization improvements. |
| Approach: | They revisit the neural probabilistic language model (NPLM) of Bengio et al. (2003) which simply concatenates word embeddings within a fixed window and passes the result through a feed-forward network to predict the next word. |
| Outcome: | The proposed model performs better on word-level language model benchmarks than a baseline Transformer with short input contexts but struggles to handle long-term dependencies. |
From Prejudice to Parity: A New Approach to Debiasing Large Language Model Word Embeddings (2025.coling-main)
Copied to clipboard
| Challenge: | Existing work in this field has looked most commonly into gender bias, racial bias, and religious bias. |
| Approach: | They propose an algorithm that uses a neural network to perform ‘soft debiasing’ and build on the seminal work of (CITATION) and (CitATION). |
| Outcome: | The proposed algorithm outperforms current methods on gender, race, and religion metrics on a wide range of metrics. |
Examining the Inductive Bias of Neural Language Models with Artificial Languages (2021.acl-long)
Copied to clipboard
| Challenge: | a novel method for investigating inductive biases of language models using artificial languages is proposed . we show that modern neural architectures used for language modeling are intrinsically black boxes . |
| Approach: | They propose a method to investigate inductive biases of language models using artificial languages . they use languages to create parallel corpora across languages that differ only in word order . |
| Outcome: | The proposed method shows that language models can be used to model a wide variety of languages. |
Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach (2020.acl-main)
Copied to clipboard
| Challenge: | incorporating syntactic structure into language models has been a challenge since the 1990s. |
| Approach: | They propose to use syntactic information to integrate syntastic structure into neural language models by providing ground truth parse trees as additional training signals. |
| Outcome: | The proposed model achieves lower perplexity and better quality when ground truth parse trees are provided as training signals. |