LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better (P18-1)
Copied to clipboard
| Challenge: | a recent study found that language models fail to learn long-range syntax sensitive dependencies. |
| Approach: | They propose to use a subject-verb agreement diagnostic to determine whether language models can learn long-range syntax sensitive dependencies. |
| Outcome: | The proposed model outperforms left-corner and bottom-up variants in learning non-local dependencies. |
Similar Papers
The emergence of number and syntax units in LSTM language models (N19-1)
Copied to clipboard
| Challenge: | a recent study shows that LSTMs can capture syntax-sensitive generalizations such as long-distance number agreement. |
| Approach: | They investigate the inner mechanics of number tracking in LSTMs at the single neuron level . they find that long-distance number information is largely managed by two "number units" importantly, the behaviour of these units is partially controlled by other units to track syntactic structure . |
| Outcome: | The proposed model is based on a language model with a long-distance number agreement task. |
Structural Supervision Improves Learning of Non-Local Grammatical Dependencies (N19-1)
Copied to clipboard
| Challenge: | State-of-the-art LSTM language models learn sequential contingencies with some success . LS models fail to learn other non-local grammatical dependencies, however . |
| Approach: | They compare LSTM language models with RNNGs to examine grammatical dependencies . they find that hierarchical supervision improves learning of non-local dependencies. |
| Outcome: | The proposed model outperforms the existing model on non-local dependencies and learns many of the Island Constraints on the filler-gap dependency. |
Representations of Syntax [MASK] Useful: Effects of Constituency and Dependency Structure in Recursive LSTMs (2020.acl-main)
Copied to clipboard
| Challenge: | a constituency-based network generalizes more robustly than a dependency-based one . a sequential LSTM can learn the appropriate rules governing these dependencies . |
| Approach: | They evaluate whether constituency-based networks introduce biases for syntactic structure . they find that constituency networks generalize more robustly than dependency networks . |
| Outcome: | The proposed model generalizes more robustly than a dependency-based model, the study shows . it shows that data augmentation can improve the robustness of the model on small data sets. |
LSTMs Compose—and Learn—Bottom-Up (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in NLP shows that LSTMs capture compositional structure in language data. |
| Approach: | They propose to measure the decompositional interdependence between word meanings in an LSTM based on their gate interactions. |
| Outcome: | The proposed model can model syntactic relationships rather than learning the longer-range relations independently. |
Revisiting the Hierarchical Multiscale LSTM (C18-1)
Copied to clipboard
| Challenge: | Hierarchical Multiscale LSTM model learns structure from character input . high complexity of architecture, training and implementations might hinder its applicability . |
| Approach: | They propose to reproduce and ablate hierarchical multiscale LSTM language model and show that simplifying certain aspects of the architecture can improve its performance. |
| Outcome: | The proposed model performs better when simplified and linguistic units are learned by different levels of the model. |
Scalable Syntax-Aware Language Models Using Knowledge Distillation (P19-1)
Copied to clipboard
| Challenge: | Prior work has shown that syntactic neural language models learn from small amounts of training data more effectively than sequential models. |
| Approach: | They propose a knowledge distillation technique that transfers knowledge from a syntactic language model trained on a small corpus to an LSTM language model and enables it to develop a more structurally sensitive representation of the larger training data. |
| Outcome: | The proposed method improves on baseline syntactic evaluations on LSTMs with a higher level of accuracy than previous methods. |
Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that unsupervised sentence representations of neural networks encode syntactic information by observing that neural language models are able to predict the agreement between a verb and its subject. |
| Approach: | They propose to take an alternative look at these results by studying whether neural networks are able to build an abstract sentence representation rather than capture surface statistical regularities. |
| Outcome: | The proposed model can achieve high accuracy on the long-range French object-verb agreement, indicating a possible flaw in the model's syntactic ability. |
Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have not investigated the relationship between a token's frequency in the training corpus and syntactic properties models learn about it. |
| Approach: | They develop controlled experiments that probe models’ syntactic nominal number and verbal argument structure generalizations for tokens seen as few as two times during training. |
| Outcome: | The proposed models can make syntactic generalizations for tokens seen as few as two times during training and transfer them to transformed contexts. |
RNN Simulations of Grammaticality Judgments on Long-distance Dependencies (C18-1)
Copied to clipboard
| Challenge: | LSTM networks can detect linguistic structures which are ungrammatical due to extraction violations, but are sensitive to linguistic processing factors. |
| Approach: | They propose to use LSTM networks to detect ungrammatical sentences by detecting extra arguments and subject-relative clause island violations. |
| Outcome: | The proposed model can correctly classify (un)grammatical sentences, in certain conditions, but is sensitive to linguistic processing factors and unable to induce a more abstract notion of grammaticality. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |