Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT (2022.coling-1)
Copied to clipboard
| Challenge: | Existing research shows that humans are prone to making agreement errors with specific constructions. |
| Approach: | They compare the performance of BERT-base and that of humans using crowdsourcing . they find that meaningfulness is stronger for BERT than for humans . |
| Outcome: | The proposed model performs better than humans on a crowdsourcing experiment . |
Similar Papers
The Functional Relevance of Probed Information: A Case Study (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that transformer models like BERT rely on number information encoded in their representations of sentences’ subjects and head verbs when performing subject-verb agreement. |
| Approach: | They propose to use probing to find out which words contain functionally relevant information encoded in the representations of subject plurality and words that agree with it in number in BERT. |
| Outcome: | The proposed model only uses the subject plurality information encoded in its representations of the subject and words that agree with it in number. |
Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task (2022.findings-acl)
Copied to clipboard
| Challenge: | lexically-independent subject-verb number agreement (NA) is performed by transformer-based neural language models (NLMs) . but when as little as one attractor is present, the model fails to perform lexical generalization . |
| Approach: | They propose to disrupt lexical patterns found in naturally occurring stimuli for each targeted structure in a novel fine-grained analysis of BERT's behavior. |
| Outcome: | The proposed model generalizes well for simple templates, but fails to perform lexically-independent generalization when as little as one attractor is present. |
A Simple and Robust Approach to Detecting Subject-Verb Agreement Errors (N19-1)
Copied to clipboard
| Challenge: | a recent study shows that neural sequential labelers overfit their training data to detect SVA errors. |
| Approach: | They propose a simple protocol that generates a neural sequential labeler from silver standard data and gold standard data. |
| Outcome: | The proposed method leads to more robust detection of SVA errors on silver standard data and gold standard data. |
Frequency Effects on Syntactic Rule Learning in Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models perform well on a variety of linguistic tasks that require symbolic reasoning, raising the question of whether such models implicitly represent abstract symbols and rules. |
| Approach: | They investigate the performance of BERT on English subject–verb agreement by analyzing word frequency and absolute frequency of verb forms. |
| Outcome: | The proposed model generalizes well to subject–verb pairs that never occurred in training, suggesting a degree of rule-governed behavior. |
Different types of syntactic agreement recruit the same units within large language models (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models can reliably distinguish grammatical from ungrammatically sentences, but how gramatical knowledge is represented within the models remains an open question. |
| Approach: | They use a functional localization approach inspired by cognitive neuroscience to identify the LLM units most responsive to 67 English syntactic phenomena in seven open-weight models. |
| Outcome: | The proposed model is most responsive to 67 English syntactic phenomena and consistently supports model performance. |
The Curious Case of Control (2022.emnlp-main)
Copied to clipboard
| Challenge: | Normally-developing children struggle with subject control clauses long after they have acquired the components to understand them. |
| Approach: | They examine whether heuristics based on semantic roles are consistent with children's English . they find that models are more sensitive to agent-patient information . |
| Outcome: | The results show that models fail on subject control but fail on object control . the authors show that raising salience of agent and patient relations results in significant changes . |
Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that unsupervised sentence representations of neural networks encode syntactic information by observing that neural language models are able to predict the agreement between a verb and its subject. |
| Approach: | They propose to take an alternative look at these results by studying whether neural networks are able to build an abstract sentence representation rather than capture surface statistical regularities. |
| Outcome: | The proposed model can achieve high accuracy on the long-range French object-verb agreement, indicating a possible flaw in the model's syntactic ability. |
Assessing the influence of attractor-verb distance on grammatical agreement in humans and language models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a subject-verb agreement in the presence of an attractor noun is a complex behavior . formal linguistic theories postulate the existence of an underlying structure that governs language processing . |
| Approach: | They hypothesize that the attractor-verb agreement may be a factor in grammatical decision-making . they hypothesized that classical models of attraction might suffice to explain this phenomenon . |
| Outcome: | The proposed model improves on humans and artificial neural networks while keeping the length of the sentence equal. |
Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages (N19-1)
Copied to clipboard
| Challenge: | Recent studies have identified both strengths and limitations of recurrent neural networks (RNNs) in applied natural language processing tasks. |
| Approach: | They propose a paradigm that addresses typological differences between languages . they create synthetic versions of English and train them to predict agreement features . |
| Outcome: | The proposed model improves on predicting agreement with subject and object, suggesting that RNNs have a recency bias. |
He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing models for factuality prediction are lacking for English . Traditionally, event factualism is triggered by fixed properties of lexical items . |
| Approach: | They propose a model that exploits common surface patterns that correlate with factuality labels. |
| Outcome: | The proposed model achieves the best performance on four factuality datasets. |