Papers with RNNG
Neural language models as psycholinguistic subjects: Representations of syntactic state (N19-1)
Copied to clipboard
| Challenge: | a recent study examines the extent to which neural network language models reflect incremental representations of syntactic state . we examine neural network model behavior on sentences chosen to probe specific aspects of the learned representations . |
| Approach: | They employ experimental methodologies developed in psycholinguistics to study syntactic representation in the human mind. |
| Outcome: | The proposed models are trained on large datasets and only sensitive to subtle cues . the results raise questions about the accuracy of the models and their performance . |
Unsupervised Recurrent Neural Network Grammars (N19-1)
Copied to clipboard
| Challenge: | RNNGs model syntax and structure by incrementally generating a syntax tree and sentence in a top-down, left-to-right order. |
| Approach: | They explore unsupervised learning of recurrent neural network grammars for language modeling and grammar induction. |
| Outcome: | The proposed model outperforms standard sequential language models and improves parsing performance. |
How Much Syntactic Supervision is “Good Enough”? (2023.findings-eacl)
Copied to clipboard
| Challenge: | RNNGs with syntactic supervision underperformed RNNs with some syntaktic supervision, whereas RNNS with mild supervision achieved the best performance comparable to the state-of-the-art GPT-2-XL. |
| Approach: | They propose a method where syntactic LMs are gradually ablated from full syntatic supervision to zero syntastic supervision by preserving NP, VP, PP, SBAR nonterminal symbols. |
| Outcome: | The proposed method underperforms the RNNGs with zero syntactic supervision, and the LMs with mild syntaktic supervision perform better than the state-of-the-art GPT-2-XL. |
On Importance Sampling-Based Evaluation of Latent Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to evaluate language models using latent structures are intractable as they require marginalizing over the latent space. |
| Approach: | They propose to use importance sampling to evaluate latent language models . they elucidate subtle differences in how importance sampling is applied . |
| Outcome: | The proposed model performs better on tasks requiring structure and interpretability. |
Neural Generative Rhetorical Structure Parsing (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to RST parsing use discriminative models, but these are less sample efficient and datasets are typically small. |
| Approach: | They propose a document-level RNN grammar with a bottom-up traversal order and a beam search algorithm that keeps track of both structure-and word-generating actions without exhibit-ing this branching bias. |
| Outcome: | The proposed model outperforms a discriminative model with the same features by 2.6 F1points and achieves comparable performance to the state-of-the-art. |
Finding syntax in human encephalography with beam search (P18-1)
Copied to clipboard
| Challenge: | RNNGs are generative models of (tree , string ) pairs that evaluate derivational choices . a non-syntactic neural language model yields no reliable effects . |
| Approach: | They propose to combine a probabilistic generative grammar with a parsing procedure that uses it to manage syntactic derivations as it advances from one word to the next. |
| Outcome: | The proposed model derives two amplitude effects when used against human encephalography data. |
Structural Supervision Improves Learning of Non-Local Grammatical Dependencies (N19-1)
Copied to clipboard
| Challenge: | State-of-the-art LSTM language models learn sequential contingencies with some success . LS models fail to learn other non-local grammatical dependencies, however . |
| Approach: | They compare LSTM language models with RNNGs to examine grammatical dependencies . they find that hierarchical supervision improves learning of non-local dependencies. |
| Outcome: | The proposed model outperforms the existing model on non-local dependencies and learns many of the Island Constraints on the filler-gap dependency. |