| Challenge: | lexicosyntactic inferences are triggered by surprising aspects of the syntactical context that a word occurs in. |
| Approach: | They build a factuality judgment dataset for English clause-embedding verbs in various syntactic contexts and use it to probe the behavior of current state-of-the-art neural systems. |
| Outcome: | The proposed model makes systematic errors that are visible through the lens of factuality prediction. |
Similar Papers
Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language? (2020.acl-main)
Copied to clipboard
| Challenge: | Despite the success of language models using neural networks, it remains unclear to what extent neural models have the generalization ability to perform inferences. |
| Approach: | They propose a method to evaluate whether neural models can learn systematicity of monotonicity inference in natural language. |
| Outcome: | The proposed method shows that neural models can perform inferences on unseen combinations of lexical and logical phenomena when syntactic structures are similar between training and test sets. |
Predicting Reference: What do Language Models Learn about Discourse Models? (2020.emnlp-main)
Copied to clipboard
| Challenge: | a growing literature that probes neural language models to assess their latent acquisition of grammatical knowledge has not investigated their acquisition of discourse modeling ability. |
| Approach: | They draw on a psycholinguistic literature that has established how different contexts affect referential biases concerning who is likely to be referred to next. |
| Outcome: | The proposed models do not resemble human language users, the authors show . their models capture the linguistic knowledge required to perform discourse modeling . |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
The Return of Lexical Dependencies: Neural Lexicalized PCFGs (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to grammar induction focus on discovering constituents or dependencies. |
| Approach: | They propose to model lexical dependencies using context free grammars instead of lexicals . they show that this unified framework induces both constituents and dependencies . |
| Outcome: | The proposed model overcomes sparsity problems and induces constituents and dependencies better than the current methods. |
A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on syntactic knowledge models has not provided a clear picture of the properties required to produce proper syntaktic generalizations. |
| Approach: | They propose to evaluate syntactic knowledge of language models by varying model architectures . they find substantial differences in syntaktic generalization performance by model architecture . |
| Outcome: | The proposed model architectures outperform other architectures on a set of 34 English-language syntactic test suites. |
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Targeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts. |
| Approach: | They apply causal mediation analysis to pre-trained neural language models to investigate their models' preferences for grammatical inflections and whether neurons process subject-verb agreement similarly across sentences with different syntactic structures. |
| Outcome: | The proposed model can predict correct token from grammatically minimally different continuations with high accuracy even in difficult contexts. |
Neural language models as psycholinguistic subjects: Representations of syntactic state (N19-1)
Copied to clipboard
| Challenge: | a recent study examines the extent to which neural network language models reflect incremental representations of syntactic state . we examine neural network model behavior on sentences chosen to probe specific aspects of the learned representations . |
| Approach: | They employ experimental methodologies developed in psycholinguistics to study syntactic representation in the human mind. |
| Outcome: | The proposed models are trained on large datasets and only sensitive to subtle cues . the results raise questions about the accuracy of the models and their performance . |
Interpreting Predictions of NLP Models (2020.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial will provide a background on interpretation techniques for neural NLP models. |
| Approach: | This tutorial will provide a background on interpretation techniques for NLP models . it will examine saliency maps, input perturbations, adversarial attacks and influence functions . |
| Outcome: | This tutorial will provide a background on interpretation techniques . examples-specific interpretations include saliency maps, input perturbations, adversarial attacks, influence functions . |
Deep Latent Variable Models of Natural Language (D18-3)
Copied to clipboard
| Challenge: | In this tutorial, we will discuss the challenges of applying neural variational inference to NLP problems. |
| Approach: | The tutorial will cover deep latent variable models in the case where exact inference over the latent variables is tractable. |
| Outcome: | The proposed tutorial will cover deep latent variable models in the case where inference cannot be performed tractably and when it is not . |
Deep Learning for Natural Language Inference (N19-5)
Copied to clipboard
| Challenge: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning for language understanding and reasoning. |
| Approach: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development and cutting- edge deep learning models. |
| Outcome: | This tutorial discusses cutting-edge research on NLI, including recent advance on dataset development, cutting- edge deep learning models, and highlights from recent research on using NLI to understand capabilities and limits of deep learning model for language understanding and reasoning. |