| Challenge: | Recent models have added structure to recurrent neural networks at the cost of giving up exact inference, or using soft structure instead of latent variables. |
| Approach: | They propose a syntactic generative model with exact marginalization that supports dependency parsing and language modeling. |
| Outcome: | The proposed models achieve state-of-the-art for supervised dependency parsing and language modeling. |
Similar Papers
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
Unsupervised Recurrent Neural Network Grammars (N19-1)
Copied to clipboard
| Challenge: | RNNGs model syntax and structure by incrementally generating a syntax tree and sentence in a top-down, left-to-right order. |
| Approach: | They explore unsupervised learning of recurrent neural network grammars for language modeling and grammar induction. |
| Outcome: | The proposed model outperforms standard sequential language models and improves parsing performance. |
Unsupervised Learning of Syntactic Structure with Invertible Neural Projections (D18-1)
Copied to clipboard
| Challenge: | Unsupervised learning of syntactic structure is typically performed using generative models with discrete latent variables and multinomial parameters. |
| Approach: | They propose a generative model that jointly learns discrete syntactic structure and continuous word representations in an unsupervised fashion by cascading an invertible neural network with a structured generative prior. |
| Outcome: | The proposed model outperforms state-of-the-art models on part-of speech (POS) induction and unsupervised dependency parsing without gold POS annotation. |
Dependency-based Mixture Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models to incorporate syntactic structures into neural language models have relied heavily on elaborate components for a specific language model, which makes them unwieldy in practice to fit into other models. |
| Approach: | They propose a dependency-based mixture language model that incorporates syntactic structures into neural language models by mixing previous dependency modeling probabilities with self-attention. |
| Outcome: | The proposed method can be easily and effectively applied to different neural language models while improving neural text generation on various tasks. |
How Do Neural Sequence Models Generalize? Local and Global Cues for Out-of-Distribution Prediction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Using RNN and transformer language models, we show consistent generalization in out-of-distribution contexts. |
| Approach: | They propose two idealized models of generalization in next-word prediction . they show that neural language models interpolate between these two forms of generalisation . |
| Outcome: | The proposed models exhibit consistent generalization in out-of-distribution contexts. |
Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language? (2020.acl-main)
Copied to clipboard
| Challenge: | Despite the success of language models using neural networks, it remains unclear to what extent neural models have the generalization ability to perform inferences. |
| Approach: | They propose a method to evaluate whether neural models can learn systematicity of monotonicity inference in natural language. |
| Outcome: | The proposed method shows that neural models can perform inferences on unseen combinations of lexical and logical phenomena when syntactic structures are similar between training and test sets. |
The Limitations of Limited Context for Constituency Parsing (2021.acl-long)
Copied to clipboard
| Challenge: | a language model that is syntax-aware can produce better samples, authors say . a recent study shows that neural approaches to syntax can perform unsupervised syntactic parsing . |
| Approach: | They propose to incorporate syntax into neural approaches in NLP to produce better samples . they find that the first time neural approaches were able to perform unsupervised syntactic parsing . |
| Outcome: | The proposed models can perform unsupervised syntactic parsing, but they are lagging behind . the proposed models are based on a sandbox of probabilistic context-free-grammars . |
Enhancing Unsupervised Generative Dependency Parser with Contextual Information (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to unsupervised dependency parsing are based on probabilistic generative models that learn the joint distribution of the given sentence and its parse. |
| Approach: | They propose a probabilistic model that generates a sentence and its parse from a latent representation, which encodes global contextual information of the generated sentence. |
| Outcome: | The proposed model achieves competitive accuracy compared with state-of-the-art models. |
Interpretation of NLP models through input marginalization (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to interpret NLP predictions replace each token with a predefined value, resulting in misleading interpretations. |
| Approach: | They propose to marginalize each token out of the training data distribution to demystify the "black box" property of deep neural networks for natural language processing. |
| Outcome: | The proposed method marginalizes each token out of the training data distribution. |
Deep Latent Variable Models of Natural Language (D18-3)
Copied to clipboard
| Challenge: | In this tutorial, we will discuss the challenges of applying neural variational inference to NLP problems. |
| Approach: | The tutorial will cover deep latent variable models in the case where exact inference over the latent variables is tractable. |
| Outcome: | The proposed tutorial will cover deep latent variable models in the case where inference cannot be performed tractably and when it is not . |