| Challenge: | Long short term memory units are powerful tools for language modeling, but their performance can be limited by the number of parameters. |
| Approach: | They propose a pyramidal recurrent unit which enables learning representations in high dimensional space with more generalization power and fewer parameters. |
| Outcome: | The proposed model outperforms existing models with different gating mechanisms and transformations on word-level language modeling tasks. |
Similar Papers
How LSTM Encodes Syntax: Exploring Context Vectors and Semi-Quantization on Natural Text (2020.coling-main)
Copied to clipboard
| Challenge: | LSTMs are widely used to capture informative long-term syntactic dependencies, but how they are reflected in their internal vectors for natural text has not been adequately investigated. |
| Approach: | They analyze how syntactic dependencies are reflected in LSTM's internal gates by learning a language model where syntaktic structures are implicitly given. |
| Outcome: | The proposed model can predict whether a word is inside a phrase structure or not from a small number of components of the context-update vector. |
Simple Recurrent Units for Highly Parallelizable Recurrence (D18-1)
Copied to clipboard
| Challenge: | recurrent neural networks scale poorly due to the intrinsic difficulty in parallelizing their state computations. |
| Approach: | They propose a simple recurrent unit that provides expressive recurrence and allows highly parallel implementation. |
| Outcome: | The proposed model achieves 5—9x speed-up over cuDNN-optimized LSTM on classification and question answering datasets and delivers stronger results than LS and convolutional models. |
The emergence of number and syntax units in LSTM language models (N19-1)
Copied to clipboard
| Challenge: | a recent study shows that LSTMs can capture syntax-sensitive generalizations such as long-distance number agreement. |
| Approach: | They investigate the inner mechanics of number tracking in LSTMs at the single neuron level . they find that long-distance number information is largely managed by two "number units" importantly, the behaviour of these units is partially controlled by other units to track syntactic structure . |
| Outcome: | The proposed model is based on a language model with a long-distance number agreement task. |
Lower Bounds on the Expressivity of Recurrent Neural Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies of the representational capacity of neural LMs have focused on their ability to recognize formal languages. |
| Approach: | They propose to connect recurrent neural networks (RNNs) as classifiers to finite-state automatas (FSAs) and a probabilistic FSA to characterize their representational capacity. |
| Outcome: | The proposed models can express arbitrary regular LMs with linearly bounded precision. |
Sentence-State LSTM for Text Representation (P18-1)
Copied to clipboard
| Challenge: | LSTMs have been shown to suffer from various limitations due to their sequential nature. |
| Approach: | They propose to model hidden states of all words simultaneously at each recurrent step rather than one word at a time. |
| Outcome: | The proposed model has strong representation power, giving competitive performances compared to stacked BiLSTM models with similar parameter numbers. |
On Efficiently Representing Regular Languages as RNNs (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). |
| Approach: | They generalize their construction and show that RNNs can efficiently represent a larger class of LMs than previously claimed. |
| Outcome: | The results suggest that RNNs can represent a larger class of LMs than previously claimed . |
Recurrent Neural Language Models as Probabilistic Finite-state Automata (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the expressive power of recurrent neural network LMs to recognize unweighted formal languages. |
| Approach: | They propose to model a strict subset of probabilistic finite-state automata with RNNs . they show that an RNN requires left(N ||right) neurons to represent an LM . |
| Outcome: | The proposed language models can represent a strict subset of probabilistic distributions expressed by finite-state models. |
Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | LSTMs can capture syntactic rules in artificial languages, but it is unclear whether they are as capable in natural languages. |
| Approach: | They propose a causal account of structural properties as carried by paths across gates and neurons of a recurrent neural network that localizes and segments the concept into a set of gate or neuron-level paths. |
| Outcome: | The proposed model improves on a widely-studied multi-layer LSTM language model showing that it can learn subject-verb number agreement in English. |
Neural language models as psycholinguistic subjects: Representations of syntactic state (N19-1)
Copied to clipboard
| Challenge: | a recent study examines the extent to which neural network language models reflect incremental representations of syntactic state . we examine neural network model behavior on sentences chosen to probe specific aspects of the learned representations . |
| Approach: | They employ experimental methodologies developed in psycholinguistics to study syntactic representation in the human mind. |
| Outcome: | The proposed models are trained on large datasets and only sensitive to subtle cues . the results raise questions about the accuracy of the models and their performance . |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |