| Challenge: | Recurrent neural networks (RNNs) can induce non-trivial properties of language. |
| Approach: | They investigate whether RNNs can track hierarchical syntactic structure . they include nonsensical sentences where RNN cannot rely on semantic cues . |
| Outcome: | The proposed models can predict long-distance agreement in nonsensical sentences in Italian and English. |
Similar Papers
On the Practical Ability of Recurrent Neural Networks to Recognize Hierarchical Languages (2020.coling-main)
Copied to clipboard
| Challenge: | recurrent models have been effective in NLP tasks but performance on context-free languages (CFLs) is weak. |
| Approach: | They evaluate the performance of recurrent models on Dyck-n languages . they find that they are expressive enough to recognize Dyck words of arbitrary lengths if their depths are bounded. |
| Outcome: | The proposed models generalize well on Dyck-n languages, while performing poorly on longer test strings. |
Deep RNNs Encode Soft Hierarchical Syntax (P18-2)
Copied to clipboard
| Challenge: | Existing studies show that syntactic information is useful for a wide variety of NLP tasks. |
| Approach: | They propose to use word-level representations to learn internal representations that capture soft hierarchical notions of syntax from highly varied supervision. |
| Outcome: | The proposed model encodes significant amounts of syntax even without explicit supervision. |
On Efficiently Representing Regular Languages as RNNs (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). |
| Approach: | They generalize their construction and show that RNNs can efficiently represent a larger class of LMs than previously claimed. |
| Outcome: | The results suggest that RNNs can represent a larger class of LMs than previously claimed . |
Priorless Recurrent Networks Learn Curiously (2020.coling-main)
Copied to clipboard
| Challenge: | a recent study shows domain-general recurrent neural networks reproduce human language behaviours . a lack of a unified concept of number agreement between these processes is a limitation of the model . |
| Approach: | They propose to use domain-general recurrent neural networks without explicit linguistic inductive biases to reproduce human language behaviours. |
| Outcome: | The proposed model can learn number agreement within unnatural sentences, the authors show . they show that the model has an effective understanding of singular versus plural for individual sentences . |
RNNs can generate bounded hierarchical languages with optimal memory (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that RNNs can efficiently generate bounded hierarchical languages with high syntactic fidelity, but their success is not well-understood theoretically. |
| Approach: | They propose a language of well-nested brackets and m-bounded nesting depth . they prove that an RNN with O(m log k) hidden units suffices, an exponential reduction in memory, by an explicit construction. |
| Outcome: | The proposed language is well-nested brackets and has m-bounded nesting depth . it shows that an RNN with O(m log k) hidden units suffices, an exponential reduction in memory, by an explicit construction. |
The Importance of Being Recurrent for Modeling Hierarchical Structure (D18-1)
Copied to clipboard
| Challenge: | Recent work shows that recurrent neural networks can implicitly capture hierarchical information when trained to solve common natural language processing tasks. |
| Approach: | They propose a convolutional sequence-to-sequence model that exploits hierarchical information implicitly. |
| Outcome: | The proposed model is recurrent and non-recurrent, and it can model hierarchical structure implicitly. |
Recurrent Neural Networks with Mixed Hierarchical Structures and EM Algorithm for Natural Language Processing (2022.lrec-1)
Copied to clipboard
| Challenge: | A variety of hierarchical RNN models have been proposed to incorporate hierarchically-based hierarchic information in modeling languages in the literature. |
| Approach: | They propose a latent indicator layer approach to identify and learn hierarchical information and develop an EM algorithm to handle the latent indicators layer in training. |
| Outcome: | The proposed approach outperforms other RNN-based models in document classification tasks. |
RNN Simulations of Grammaticality Judgments on Long-distance Dependencies (C18-1)
Copied to clipboard
| Challenge: | LSTM networks can detect linguistic structures which are ungrammatical due to extraction violations, but are sensitive to linguistic processing factors. |
| Approach: | They propose to use LSTM networks to detect ungrammatical sentences by detecting extra arguments and subject-relative clause island violations. |
| Outcome: | The proposed model can correctly classify (un)grammatical sentences, in certain conditions, but is sensitive to linguistic processing factors and unable to induce a more abstract notion of grammaticality. |
A Formal Hierarchy of RNN Architectures (2020.acl-main)
Copied to clipboard
| Challenge: | Existing theories of expressive power of RNNs are limited. |
| Approach: | They propose a formal hierarchy of the expressive capacity of RNN architectures based on two formal properties: space complexity and rational recurrence. |
| Outcome: | The proposed model is based on the theory of “saturated” RNNs and shows that it obeys a similar hierarchy to unsaturated RNN models. |
Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammars (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature is agnostic about a parsing strategy of hierarchical models . a recent study showed that hierarchically model hierarchic structures capture grammatical dependencies much better than RNNs in targeted syntactic evaluations. |
| Approach: | They evaluated three LMs with head-final left-branching structures and Recurrent Neural Network Grammars with top-down and left-corner parsing strategies as hierarchical models. |
| Outcome: | The proposed model outperforms top-down and left-corner models against human reading times in Japanese. |