How much complexity does an RNN architecture need to learn syntax-sensitive dependencies? (2020.acl-srw)
Copied to clipboard
| Challenge: | Long-term memory (LSTM) networks are capable of encapsulating long-range dependencies . but simple recurrent networks (SRNs) have been less successful at capturing long-term dependencies and loci of grammatical errors in an unsupervised setting. |
| Approach: | They propose a new architecture that incorporates the decaying nature of neuronal activations and models the excitatory and inhibitory connections in a population of neurons. |
| Outcome: | The proposed architecture shows competitive performance relative to LSTMs on subject-verb agreement, sentence grammaticality, and language modeling tasks. |
Similar Papers
RNN Simulations of Grammaticality Judgments on Long-distance Dependencies (C18-1)
Copied to clipboard
| Challenge: | LSTM networks can detect linguistic structures which are ungrammatical due to extraction violations, but are sensitive to linguistic processing factors. |
| Approach: | They propose to use LSTM networks to detect ungrammatical sentences by detecting extra arguments and subject-relative clause island violations. |
| Outcome: | The proposed model can correctly classify (un)grammatical sentences, in certain conditions, but is sensitive to linguistic processing factors and unable to induce a more abstract notion of grammaticality. |
On Efficiently Representing Regular Languages as RNNs (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). |
| Approach: | They generalize their construction and show that RNNs can efficiently represent a larger class of LMs than previously claimed. |
| Outcome: | The results suggest that RNNs can represent a larger class of LMs than previously claimed . |
A Formal Hierarchy of RNN Architectures (2020.acl-main)
Copied to clipboard
| Challenge: | Existing theories of expressive power of RNNs are limited. |
| Approach: | They propose a formal hierarchy of the expressive capacity of RNN architectures based on two formal properties: space complexity and rational recurrence. |
| Outcome: | The proposed model is based on the theory of “saturated” RNNs and shows that it obeys a similar hierarchy to unsaturated RNN models. |
Lower Bounds on the Expressivity of Recurrent Neural Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies of the representational capacity of neural LMs have focused on their ability to recognize formal languages. |
| Approach: | They propose to connect recurrent neural networks (RNNs) as classifiers to finite-state automatas (FSAs) and a probabilistic FSA to characterize their representational capacity. |
| Outcome: | The proposed models can express arbitrary regular LMs with linearly bounded precision. |
On the Practical Computational Power of Finite Precision RNNs for Language Recognition (P18-2)
Copied to clipboard
| Challenge: | Recurrent Neural Networks (RNNs) are famously known to be Turing complete, but this relies on infinite precision in the states and unbounded computation time. |
| Approach: | They propose to use LSTM and Elman-RNN with ReLU activation to study RNNs . they show that LS and ReLU-RNns can easily implement counting behavior . |
| Outcome: | The LSTM and the Elman-RNN with ReLU activation are stronger than the RNN with squashing activation and the GRU. |
How LSTM Encodes Syntax: Exploring Context Vectors and Semi-Quantization on Natural Text (2020.coling-main)
Copied to clipboard
| Challenge: | LSTMs are widely used to capture informative long-term syntactic dependencies, but how they are reflected in their internal vectors for natural text has not been adequately investigated. |
| Approach: | They analyze how syntactic dependencies are reflected in LSTM's internal gates by learning a language model where syntaktic structures are implicitly given. |
| Outcome: | The proposed model can predict whether a word is inside a phrase structure or not from a small number of components of the context-update vector. |
What Part of the Neural Network Does This? Understanding LSTMs by Measuring and Dissecting Neurons (D19-1)
Copied to clipboard
| Challenge: | Biological neural systems consist of a huge number of neurons, and can react to the environment in complicated ways. |
| Approach: | They propose a metric to quantify the sensitivity of neurons to each label and conduct experiments to prove it. |
| Outcome: | The proposed metric is based on a set of experiments that show that dropping an arbitrary neuron significantly degrades the accuracy of the model. |
Colorless Green Recurrent Networks Dream Hierarchically (N18-1)
Copied to clipboard
| Challenge: | Recurrent neural networks (RNNs) can induce non-trivial properties of language. |
| Approach: | They investigate whether RNNs can track hierarchical syntactic structure . they include nonsensical sentences where RNN cannot rely on semantic cues . |
| Outcome: | The proposed models can predict long-distance agreement in nonsensical sentences in Italian and English. |
Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages (N19-1)
Copied to clipboard
| Challenge: | Recent studies have identified both strengths and limitations of recurrent neural networks (RNNs) in applied natural language processing tasks. |
| Approach: | They propose a paradigm that addresses typological differences between languages . they create synthetic versions of English and train them to predict agreement features . |
| Outcome: | The proposed model improves on predicting agreement with subject and object, suggesting that RNNs have a recency bias. |
Inducing Grammar from Long Short-Term Memory Networks by Shapley Decomposition (2020.acl-srw)
Copied to clipboard
| Challenge: | a recent study shows that modern neural networks understand sentences implicitly by inducing recursive structures. |
| Approach: | They propose to explicitly induce grammar by tracing the computational process of a long short-term memory network. |
| Outcome: | The proposed model can explicitly induce grammar without external knowledge . tracing the computational process of a long short-term memory network is shown to be effective . |