Using Large Corpus N-gram Statistics to Improve Recurrent Neural Language Models (N19-1)
Copied to clipboard
| Challenge: | a technique that uses large corpus n-gram statistics as a regularizer for training a neural network LM on a smaller corpus is effective, and more time-efficient than training on ngrams. |
| Approach: | They propose a technique that uses large corpus n-gram statistics as a regularizer for training on a smaller corpus. |
| Outcome: | The proposed technique is effective and more time-efficient than training on a larger corpus. |
Similar Papers
Residual Learning of Neural Text Generation with n-gram Language Model (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that n-gram models can achieve satisfactory performance on a large proportion of testing cases. |
| Approach: | They propose to learn a neural LM that fits the residual between an n-gram LM and the real-data distribution. |
| Outcome: | The proposed model achieves additional performance gains over popular standalone models on three typical language tasks. |
Show Some Love to Your n-grams: A Bit of Progress and Stronger n-gram Language Modeling Baselines (N19-1)
Copied to clipboard
| Challenge: | Experimental results show that standalone n-gram models lend themselves as natural choices for resource-lean or morphologically rich languages. |
| Approach: | They run experiments on 50 languages covering all morphological language families to compare n-gram models with lstm models. |
| Outcome: | The proposed extension outperforms an lstm language model on 42 languages while its extension which explicitly injects linguistic knowledge outperformed the character-aware neural model on 8 languages. |
Estimating Marginal Probabilities of n-grams for Recurrent Neural Language Models (D18-1)
Copied to clipboard
| Challenge: | Recurrent neural network language models (RNNs) only estimate probabilities for complete sequences of text, whereas some applications require context-independent phrase probabilities instead. |
| Approach: | They propose a method to alter the RNNLM training to make it more accurate at marginal estimation. |
| Outcome: | The proposed method is effective compared to baselines including the traditional RNNLM probability and importance sampling approach. |
Quantity doesn’t buy quality syntax with neural language models (D19-1)
Copied to clipboard
| Challenge: | Recurrent neural network language models can learn to predict upcoming words with remarkably low perplexity . but in syntactically complex contexts, they often assign unexpectedly high probabilities to ungrammatical words . |
| Approach: | They investigate whether recurrent neural networks can learn to predict upcoming words with remarkably low perplexity. |
| Outcome: | The proposed models perform worse than GPT and BERT in some constructions than LSTMs in other contexts. |
Efficient Training of Language Models with Compact and Consistent Next Token Distributions (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train language models have focused on maximizing the likelihood of the next token . however, the construction and querying of such n-grams can be costly and impede training speed. |
| Approach: | They propose a method to train language models faster by pre-aggregating corpus with collapsed n-gram distribution. |
| Outcome: | The proposed model improves model quality and convergence rate while reducing variance across mini-batches compared to the standard next-token loss method. |
The Role of n-gram Smoothing in the Age of Neural Networks (2024.naacl-long)
Copied to clipboard
| Challenge: | n-gram smoothing techniques were used to overcome overfitting problems in neural language models for decades. |
| Approach: | They propose to convert any n-gram smoothing technique into a regularizer compatible with neural language models. |
| Outcome: | The proposed regularizers outperform label smoothing on language modeling and machine translation. |
How Important is a Language Model for Low-resource ASR? (2024.findings-acl)
Copied to clipboard
| Challenge: | Using an n-gram language model in ASR may seem obvious, but its absence in most implementations suggests otherwise. |
| Approach: | They examine whether using an n-gram language model in ASR can improve accuracy in low-resource languages. |
| Outcome: | The proposed model is absent in most implementations, but it does improve accuracy in English and Mandarin. |
Trained on 100 million words and still in shape: BERT meets British National Corpus (2023.findings-eacl)
Copied to clipboard
| Challenge: | masked language models are trained on ever larger corpora, but pre-training on a modestly-sized but representative, well-balanced, and publicly available corpus can reach better performance than the original BERT model. |
| Approach: | They propose an optimized LM architecture called LTG-BERT that can be used to train a competitive language model on a small and standardizable corpus. |
| Outcome: | The proposed architecture outperforms the original English BERT model on a representative, well-balanced and publicly available corpus. |
Sampling Informative Training Data for RNN Language Models (P18-3)
Copied to clipboard
| Challenge: | Existing methods for selecting training data for recurrent neural network language models are limited by computational costs. |
| Approach: | They propose an unsupervised importance sampling approach to selecting training data for recurrent neural network (RNN) language models. |
| Outcome: | The proposed approach outperforms models trained on the Billion Word and Wikitext-103 benchmark corpora. |
Lower Bounds on the Expressivity of Recurrent Neural Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies of the representational capacity of neural LMs have focused on their ability to recognize formal languages. |
| Approach: | They propose to connect recurrent neural networks (RNNs) as classifiers to finite-state automatas (FSAs) and a probabilistic FSA to characterize their representational capacity. |
| Outcome: | The proposed models can express arbitrary regular LMs with linearly bounded precision. |