An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Neural language models are often trained on positive examples, but recent studies suggest they are not robust enough to handle complex syntactic constructions. |
| Approach: | They propose to use negative examples to boost models' robustness on English sentences with a negligible loss of perplexity. |
| Outcome: | The proposed model is robust to negative examples in English with negligible loss of perplexity . |
Similar Papers
Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment (2020.acl-main)
Copied to clipboard
| Challenge: | Language modeling is widely used as pretraining for many tasks involving language processing. |
| Approach: | They extend a standard approach to evaluating language models to cases of valid interpretations . they compare model performance in English and Spanish to show that non-linguistic bias overlaps with syntactic structure in English but not Spanish. |
| Outcome: | The proposed model performance in English and Spanish shows that non-linguistic bias overlaps with syntactic structure in English but not in Spanish. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
Evaluating and Enhancing the Robustness of Neural Network-based Dependency Parsing Models with Adversarial Examples (2020.acl-main)
Copied to clipboard
| Challenge: | Previously studies focused on semantic tasks such as sentiment analysis, question answering and reading comprehension. |
| Approach: | They propose two approaches to study where and how adversarial examples exist in dependency parsing . they use a state-of-the-art parser to find adversarials in existing texts . |
| Outcome: | The proposed approaches show that adversarial examples exist in dependency parsing . they show that up to 77% of input examples admit adversarials . |
Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that unsupervised sentence representations of neural networks encode syntactic information by observing that neural language models are able to predict the agreement between a verb and its subject. |
| Approach: | They propose to take an alternative look at these results by studying whether neural networks are able to build an abstract sentence representation rather than capture surface statistical regularities. |
| Outcome: | The proposed model can achieve high accuracy on the long-range French object-verb agreement, indicating a possible flaw in the model's syntactic ability. |
End-to-End Bias Mitigation by Modelling Biases in Corpora (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that strong natural language understanding models are prone to relying on unwanted dataset biases without learning the underlying task. |
| Approach: | They propose two learning strategies to train neural models that are more robust to dataset biases and transfer better to out-of-domain datasets. |
| Outcome: | The proposed methods improve robustness in all settings and transfer better to out-of-domain datasets. |
A Reinforced Generation of Adversarial Examples for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Neural machine translation systems fail on less decent inputs, which may harm the credibility of these systems. |
| Approach: | They propose a paradigm that generates adversarial examples using reinforcement learning to expose pitfalls for a given performance metric. |
| Outcome: | The proposed paradigm produces stable attacks with meaning-preserving adversarial examples. |
Knowing False Negatives: An Adversarial Training Method for Distantly Supervised Relation Extraction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for relation extraction ignore the incompleteness of existing knowledge bases . current methods are too weak and cause noises when training and testing are not based on training data. |
| Approach: | They propose a method to automatically align unstructured text with relation instances in a knowledge base . they use heuristics to leverage the memory mechanism of deep neural networks to find out possible FN samples . |
| Outcome: | Experiments on two wildly-used benchmark datasets show the effectiveness of the proposed method. |
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Targeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts. |
| Approach: | They apply causal mediation analysis to pre-trained neural language models to investigate their models' preferences for grammatical inflections and whether neurons process subject-verb agreement similarly across sentences with different syntactic structures. |
| Outcome: | The proposed model can predict correct token from grammatically minimally different continuations with high accuracy even in difficult contexts. |
A Closer Look into the Robustness of Neural Dependency Parsers Using Better Adversarial Examples (2021.findings-acl)
Copied to clipboard
| Challenge: | Neural network-based models have been successful in a wide range of NLP tasks, but their performance is undermined by adversarial examples that would pose no confusion for humans. |
| Approach: | They propose a method to generate high-quality adversarial examples with a higher number of candidate generators and stricter filters and then verify their quality using automatic and human evaluations. |
| Outcome: | The proposed method improves the robustness of English parsing models by relying on adversarial training and model ensembling. |
What if This Modified That? Syntactic Interventions with Counterfactual Embeddings (2021.findings-acl)
Copied to clipboard
| Challenge: | Prior art aims to uncover meaningful properties within model representations, but it is unclear how faithfully such probes portray information that the models actually use. |
| Approach: | They propose a technique for generating counterfactual embeddings within models . they produce evidence that some models use a tree-distancelike representation of syntax . |
| Outcome: | The proposed technique produces evidence that some models use tree-distancelike representations of syntax in downstream prediction tasks. |