Investigating representations of verb bias in neural language models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Languages typically provide more than one grammatical construction to express certain types of messages. |
| Approach: | They propose a large benchmark dataset containing 50K human judgments for 5K distinct sentence pairs in the English dative alternation. |
| Outcome: | The proposed model outperforms recurrent architectures even under comparable parameter and training settings. |
Similar Papers
Examining the Inductive Bias of Neural Language Models with Artificial Languages (2021.acl-long)
Copied to clipboard
| Challenge: | a novel method for investigating inductive biases of language models using artificial languages is proposed . we show that modern neural architectures used for language modeling are intrinsically black boxes . |
| Approach: | They propose a method to investigate inductive biases of language models using artificial languages . they use languages to create parallel corpora across languages that differ only in word order . |
| Outcome: | The proposed method shows that language models can be used to model a wide variety of languages. |
Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages (N19-1)
Copied to clipboard
| Challenge: | Recent studies have identified both strengths and limitations of recurrent neural networks (RNNs) in applied natural language processing tasks. |
| Approach: | They propose a paradigm that addresses typological differences between languages . they create synthetic versions of English and train them to predict agreement features . |
| Outcome: | The proposed model improves on predicting agreement with subject and object, suggesting that RNNs have a recency bias. |
Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that unsupervised sentence representations of neural networks encode syntactic information by observing that neural language models are able to predict the agreement between a verb and its subject. |
| Approach: | They propose to take an alternative look at these results by studying whether neural networks are able to build an abstract sentence representation rather than capture surface statistical regularities. |
| Outcome: | The proposed model can achieve high accuracy on the long-range French object-verb agreement, indicating a possible flaw in the model's syntactic ability. |
How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases (2023.acl-long)
Copied to clipboard
| Challenge: | a recent study found that pre-training can teach language models to rely on hierarchical syntactic features . aaron ramirez: we find that pretraining on simpler language induces a hierarchic bias . |
| Approach: | They find that pre-training can teach language models to rely on hierarchical syntactic features . authors: this suggests that in cognitively plausible language acquisition settings, models may be more data-efficient . |
| Outcome: | a recent study shows that pre-training can teach language models to rely on hierarchical features . the findings suggest that in plausible language acquisition settings, language models may be more data-efficient than previously thought . |
Do Neural Language Models Overcome Reporting Bias? (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models can overcome reporting bias by estimating the plausibility of rare but unspoken facts. |
| Approach: | They revisit the experiments conducted by Gordon and Van Durme (2013) . they find that pre-trained language models overestimate the very rare . |
| Outcome: | The proposed approach overestimates the rare at the expense of the rare, while minimizing reporting bias. |
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Targeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts. |
| Approach: | They apply causal mediation analysis to pre-trained neural language models to investigate their models' preferences for grammatical inflections and whether neurons process subject-verb agreement similarly across sentences with different syntactic structures. |
| Outcome: | The proposed model can predict correct token from grammatically minimally different continuations with high accuracy even in difficult contexts. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
Do Neural Language Models Show Preferences for Syntactic Formalisms? (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work on interpretability of deep neural language models concludes that many properties of natural language syntax are encoded in their representational spaces. |
| Approach: | They propose to examine whether syntactic structure adheres to a surface-syntactical or deep syntaktic style of analysis. |
| Outcome: | The proposed model prefers Universal Dependencies (UD) over Surface-Syntactic Universal Dependency (SUD) with interesting variations across languages and layers. |
A Closer Look at Data Bias in Neural Extractive Summarization Models (D19-54)
Copied to clipboard
| Challenge: | In this paper, we examine the generalization behaviour of summarization models . we propose several properties of datasets that matter for generalization . |
| Approach: | They propose several properties of datasets which matter for generalization of summarization models. |
| Outcome: | The proposed approach improves the state-of-the-art model by rethinking the model design process on a typical dataset. |
Representation biases in sentence transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | argued that transformer-based models are not well suited for sentence-level downstream tasks. |
| Approach: | They propose to use sentence transformers to produce full-sentence representations . they propose to combine transformers with a training regime that embeds tokens into the model . |
| Outcome: | The proposed model performs better on downstream tasks than the vanilla model and its variants. |