Structural Supervision Improves Learning of Non-Local Grammatical Dependencies (N19-1)
Copied to clipboard
| Challenge: | State-of-the-art LSTM language models learn sequential contingencies with some success . LS models fail to learn other non-local grammatical dependencies, however . |
| Approach: | They compare LSTM language models with RNNGs to examine grammatical dependencies . they find that hierarchical supervision improves learning of non-local dependencies. |
| Outcome: | The proposed model outperforms the existing model on non-local dependencies and learns many of the Island Constraints on the filler-gap dependency. |
Similar Papers
Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has shown that structural supervision helps English language models learn generalizations about syntactic phenomena such as subject-verb agreement. |
| Approach: | They train LSTMs, Recurrent Neural Network Grammars, Transformer language models, and Transformer-parameterized generative parsing models on Mandarin Chinese datasets. |
| Outcome: | The proposed models learn aspects of Mandarin Chinese grammar that assess syntactic and semantic relationships. |
LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better (P18-1)
Copied to clipboard
| Challenge: | a recent study found that language models fail to learn long-range syntax sensitive dependencies. |
| Approach: | They propose to use a subject-verb agreement diagnostic to determine whether language models can learn long-range syntax sensitive dependencies. |
| Outcome: | The proposed model outperforms left-corner and bottom-up variants in learning non-local dependencies. |
Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have not investigated the relationship between a token's frequency in the training corpus and syntactic properties models learn about it. |
| Approach: | They develop controlled experiments that probe models’ syntactic nominal number and verbal argument structure generalizations for tokens seen as few as two times during training. |
| Outcome: | The proposed models can make syntactic generalizations for tokens seen as few as two times during training and transfer them to transformed contexts. |
Representations of Syntax [MASK] Useful: Effects of Constituency and Dependency Structure in Recursive LSTMs (2020.acl-main)
Copied to clipboard
| Challenge: | a constituency-based network generalizes more robustly than a dependency-based one . a sequential LSTM can learn the appropriate rules governing these dependencies . |
| Approach: | They evaluate whether constituency-based networks introduce biases for syntactic structure . they find that constituency networks generalize more robustly than dependency networks . |
| Outcome: | The proposed model generalizes more robustly than a dependency-based model, the study shows . it shows that data augmentation can improve the robustness of the model on small data sets. |
RNN Simulations of Grammaticality Judgments on Long-distance Dependencies (C18-1)
Copied to clipboard
| Challenge: | LSTM networks can detect linguistic structures which are ungrammatical due to extraction violations, but are sensitive to linguistic processing factors. |
| Approach: | They propose to use LSTM networks to detect ungrammatical sentences by detecting extra arguments and subject-relative clause island violations. |
| Outcome: | The proposed model can correctly classify (un)grammatical sentences, in certain conditions, but is sensitive to linguistic processing factors and unable to induce a more abstract notion of grammaticality. |
LSTMs Compose—and Learn—Bottom-Up (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in NLP shows that LSTMs capture compositional structure in language data. |
| Approach: | They propose to measure the decompositional interdependence between word meanings in an LSTM based on their gate interactions. |
| Outcome: | The proposed model can model syntactic relationships rather than learning the longer-range relations independently. |
Revisiting the Hierarchical Multiscale LSTM (C18-1)
Copied to clipboard
| Challenge: | Hierarchical Multiscale LSTM model learns structure from character input . high complexity of architecture, training and implementations might hinder its applicability . |
| Approach: | They propose to reproduce and ablate hierarchical multiscale LSTM language model and show that simplifying certain aspects of the architecture can improve its performance. |
| Outcome: | The proposed model performs better when simplified and linguistic units are learned by different levels of the model. |
Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | et al., 2018a, 2018b) show that LSTMs can transfer from non-linguistic data to natural language models with different types of abstract structure. |
| Approach: | They propose to use transfer learning to analyze encoding of grammatical structure in neural language models. |
| Outcome: | The proposed method improves test performance on natural language despite no overlap in surface form or vocabulary. |
The emergence of number and syntax units in LSTM language models (N19-1)
Copied to clipboard
| Challenge: | a recent study shows that LSTMs can capture syntax-sensitive generalizations such as long-distance number agreement. |
| Approach: | They investigate the inner mechanics of number tracking in LSTMs at the single neuron level . they find that long-distance number information is largely managed by two "number units" importantly, the behaviour of these units is partially controlled by other units to track syntactic structure . |
| Outcome: | The proposed model is based on a language model with a long-distance number agreement task. |
Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammars (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature is agnostic about a parsing strategy of hierarchical models . a recent study showed that hierarchically model hierarchic structures capture grammatical dependencies much better than RNNs in targeted syntactic evaluations. |
| Approach: | They evaluated three LMs with head-final left-branching structures and Recurrent Neural Network Grammars with top-down and left-corner parsing strategies as hierarchical models. |
| Outcome: | The proposed model outperforms top-down and left-corner models against human reading times in Japanese. |