Estimating Marginal Probabilities of n-grams for Recurrent Neural Language Models (D18-1)
Copied to clipboard
| Challenge: | Recurrent neural network language models (RNNs) only estimate probabilities for complete sequences of text, whereas some applications require context-independent phrase probabilities instead. |
| Approach: | They propose a method to alter the RNNLM training to make it more accurate at marginal estimation. |
| Outcome: | The proposed method is effective compared to baselines including the traditional RNNLM probability and importance sampling approach. |
Similar Papers
Using Large Corpus N-gram Statistics to Improve Recurrent Neural Language Models (N19-1)
Copied to clipboard
| Challenge: | a technique that uses large corpus n-gram statistics as a regularizer for training a neural network LM on a smaller corpus is effective, and more time-efficient than training on ngrams. |
| Approach: | They propose a technique that uses large corpus n-gram statistics as a regularizer for training on a smaller corpus. |
| Outcome: | The proposed technique is effective and more time-efficient than training on a larger corpus. |
You should evaluate your language model on marginal likelihood over tokenisations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Neural language models typically tokenise input text into sub-word units to achieve an open vocabulary. |
| Approach: | They propose that language models should be evaluated on their marginal likelihood over tokenisations instead. |
| Outcome: | The proposed approach is unsatisfactory and may bottleneck model out-of-domain performance. |
Sampling Informative Training Data for RNN Language Models (P18-3)
Copied to clipboard
| Challenge: | Existing methods for selecting training data for recurrent neural network language models are limited by computational costs. |
| Approach: | They propose an unsupervised importance sampling approach to selecting training data for recurrent neural network (RNN) language models. |
| Outcome: | The proposed approach outperforms models trained on the Billion Word and Wikitext-103 benchmark corpora. |
Training Hybrid Language Models by Marginalizing over Segmentations (P19-1)
Copied to clipboard
| Challenge: | Statistical language modeling is the problem of estimating a probability distribution over text data. |
| Approach: | They propose to marginalize over the segmentations efficiently to compute the true probability of a sequence. |
| Outcome: | The proposed model marginalizes over the segmentations to compute the true probability of a sequence on three datasets comprising seven languages. |
Residual Learning of Neural Text Generation with n-gram Language Model (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that n-gram models can achieve satisfactory performance on a large proportion of testing cases. |
| Approach: | They propose to learn a neural LM that fits the residual between an n-gram LM and the real-data distribution. |
| Outcome: | The proposed model achieves additional performance gains over popular standalone models on three typical language tasks. |
Revisiting Simple Neural Probabilistic Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent advances in language modeling have been driven not only by advances in neural architectures, but also through hardware and optimization improvements. |
| Approach: | They revisit the neural probabilistic language model (NPLM) of Bengio et al. (2003) which simply concatenates word embeddings within a fixed window and passes the result through a feed-forward network to predict the next word. |
| Outcome: | The proposed model performs better on word-level language model benchmarks than a baseline Transformer with short input contexts but struggles to handle long-term dependencies. |
Recurrent Neural Language Models as Probabilistic Finite-state Automata (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the expressive power of recurrent neural network LMs to recognize unweighted formal languages. |
| Approach: | They propose to model a strict subset of probabilistic finite-state automata with RNNs . they show that an RNN requires left(N ||right) neurons to represent an LM . |
| Outcome: | The proposed language models can represent a strict subset of probabilistic distributions expressed by finite-state models. |
A Natural Bias for Language Generation Models (2023.acl-short)
Copied to clipboard
| Challenge: | a standard probabilistic model for language generation has likely not yet learnt many semantic or syntactic rules of natural language, making it difficult to estimate the probability distribution over next tokens. |
| Approach: | They propose to initialise bias terms in a model's final linear layer with the log-unigram distribution and use it to output the unigram frequency statistics as prior knowledge. |
| Outcome: | The proposed method improves learning efficiency and improves overall performance. |
Direct Output Connection for a High-Rank Language Model (D18-1)
Copied to clipboard
| Challenge: | Neural network language models have played a central role in recent natural language processing advances. |
| Approach: | They propose a recurrent neural network language model that combines probability distributions computed from a final RNN layer and also middle layers. |
| Outcome: | The proposed language model achieves the best score on the Penn Treebank and WikiText-2 benchmark datasets. |
Large Margin Neural Language Model (D18-1)
Copied to clipboard
| Challenge: | Conventionally, neural language models are trained by minimizing perplexity (PPL) on grammatical sentences. |
| Approach: | They propose a large margin criterion for training neural language models by minimizing perplexity on grammatical sentences and propose enlarged margins for task-specific training. |
| Outcome: | The proposed method gains up to 1.1 WER reduction for speech recognition and 1.0 BLEU increase for machine translation. |