| Challenge: | Existing models to incorporate syntactic structures into neural language models have relied heavily on elaborate components for a specific language model, which makes them unwieldy in practice to fit into other models. |
| Approach: | They propose a dependency-based mixture language model that incorporates syntactic structures into neural language models by mixing previous dependency modeling probabilities with self-attention. |
| Outcome: | The proposed method can be easily and effectively applied to different neural language models while improving neural text generation on various tasks. |
Similar Papers
Direct Output Connection for a High-Rank Language Model (D18-1)
Copied to clipboard
| Challenge: | Neural network language models have played a central role in recent natural language processing advances. |
| Approach: | They propose a recurrent neural network language model that combines probability distributions computed from a final RNN layer and also middle layers. |
| Outcome: | The proposed language model achieves the best score on the Penn Treebank and WikiText-2 benchmark datasets. |
Bayesian Learning for Neural Dependency Parsing (N19-1)
Copied to clipboard
| Challenge: | Several approaches for dependency parsing in the small data regime have been proposed. |
| Approach: | They propose to use stochastic gradient Langevin dynamics to generate samples from the approximated posterior to overcome the computational and statistical costs of the approximate inference step. |
| Outcome: | The proposed model outperforms the biaffine model on 6 languages with less than 5k training instances and improves across five languages. |
Code-switched Language Models Using Dual RNNs and Same-Source Pretraining (D18-1)
Copied to clipboard
| Challenge: | Using recurrent neural networks to build language models for code-switched text is an important problem with implications to downstream applications such as speech recognition and machine translation. |
| Approach: | They propose a novel recurrent neural network unit with dual components that focus on each language in the code-switched text separately and a generative model estimated using the training data. |
| Outcome: | The proposed techniques yield significant reductions in perplexity on Mandarin-English task and improve on baseline models. |
Revisiting Simple Neural Probabilistic Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent advances in language modeling have been driven not only by advances in neural architectures, but also through hardware and optimization improvements. |
| Approach: | They revisit the neural probabilistic language model (NPLM) of Bengio et al. (2003) which simply concatenates word embeddings within a fixed window and passes the result through a feed-forward network to predict the next word. |
| Outcome: | The proposed model performs better on word-level language model benchmarks than a baseline Transformer with short input contexts but struggles to handle long-term dependencies. |
How to represent a word and predict it, too: Improving tied architectures for language modelling (D18-1)
Copied to clipboard
| Challenge: | Recent state-of-the-art models use word embeddings as input and output mappings instead of tied models. |
| Approach: | They propose to decouple hidden state from word embedding prediction . they extend their proposed modification to word2vec models . |
| Outcome: | The proposed architectures achieve comparable or better results compared to previous models without tying . the proposed architecture reduces parameters, enabling more compact models and faster learning. |
Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation (2023.findings-acl)
Copied to clipboard
| Challenge: | Syntactic structures were deemed essential in natural language processing . but since the deep learning revolution, NLP has been dominated by neural models that do not consider syntactical structures in their design. |
| Approach: | They propose a model that models latent representations of words in a sentence . they use a conditional random field to model latent and dependency arcs . |
| Outcome: | The proposed model performs competitively to transformers on small to medium sized datasets. |
Neural Syntactic Generative Models with Exact Marginalization (N18-1)
Copied to clipboard
| Challenge: | Recent models have added structure to recurrent neural networks at the cost of giving up exact inference, or using soft structure instead of latent variables. |
| Approach: | They propose a syntactic generative model with exact marginalization that supports dependency parsing and language modeling. |
| Outcome: | The proposed models achieve state-of-the-art for supervised dependency parsing and language modeling. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
Towards Making a Dependency Parser See (D19-1)
Copied to clipboard
| Challenge: | Eye trackers and gaze features collected from them have been recently applied to natural language processing (NLP) tasks such as part-of-speech tagging. |
| Approach: | They propose to leverage eye-tracking data in an RNN dependency parser when no aggregated or token-level gaze features are used at inference time. |
| Outcome: | The proposed model can be used to improve performance on non-gazed treebanks. |
On Efficiently Representing Regular Languages as RNNs (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). |
| Approach: | They generalize their construction and show that RNNs can efficiently represent a larger class of LMs than previously claimed. |
| Outcome: | The results suggest that RNNs can represent a larger class of LMs than previously claimed . |