| Challenge: | Normally-developing children struggle with subject control clauses long after they have acquired the components to understand them. |
| Approach: | They examine whether heuristics based on semantic roles are consistent with children's English . they find that models are more sensitive to agent-patient information . |
| Outcome: | The results show that models fail on subject control but fail on object control . the authors show that raising salience of agent and patient relations results in significant changes . |
Similar Papers
A Psycholinguistic Evaluation of Language Models’ Sensitivity to Argument Roles (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a systematic evaluation of large language models' sensitivity to argument roles is presented . a recent study shows that argument roles have a delayed impact on verb prediction in human sentence processing. |
| Approach: | They propose to replicate psycholinguistic studies on human argument role processing . they find that language models are able to distinguish verbs that appear in plausible and implausible contexts . |
| Outcome: | The proposed models are able to distinguish verbs that appear in plausible and implausible contexts, but none captures the same selective patterns that human comprehenders exhibit during real-time verb prediction. |
Evaluating Large Language Models on Controlled Generation Tasks (2023.emnlp-main)
Copied to clipboard
Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, Xuezhe Ma
| Challenge: | Recent studies have looked into the ability of large language models in various benchmark tasks, including question generation, reading comprehension, multilingual and etc. However, few studies investigate the controllability of large languages. |
| Approach: | They propose to compare large language models with state-of-the-start finetuned smaller models to find that large language model controls are comparable to smaller models. |
| Outcome: | The proposed model can meet hard constraints and perform better than state-of-the-art models. |
On the Role of Semantic Proto-roles in Semantic Analysis: What do LLMs know about agency? (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on large language models (LLMs) have not explored their capacity to reason over event structure . et al., 2015, 142: e007-e0027; eugene, 1985; Weiner, 1995; saab, 1985) focus on the role of large language model in decision-making . |
| Approach: | They propose to characterize agents via properties such as "instigation" and "volition" they also examine whether incorporating semantic proto-role labeling context improves SRL performance . |
| Outcome: | The proposed model improves in a zero-shot setting by incorporating proto-role labeling context . the results support previous work showing that LLMs underperform human annotators in complex semantic analysis. |
Dependency resolution at the syntax-semantics interface: psycholinguistic and computational insights on control dependencies (2023.acl-long)
Copied to clipboard
| Challenge: | Using psycholinguistic and computational experiments, we compare the ability of humans and several pre-trained masked language models to correctly identify control dependencies in Spanish sentences. |
| Approach: | They compare the ability of humans and several pre-trained masked language models to correctly identify control dependencies in Spanish sentences such as ‘José le prometió/ordenó a Mara ser ordenado/a’. |
| Outcome: | The models fail to identify the correct antecedent in non-adjacent dependencies, showing their reliance on linearity. |
Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that unsupervised sentence representations of neural networks encode syntactic information by observing that neural language models are able to predict the agreement between a verb and its subject. |
| Approach: | They propose to take an alternative look at these results by studying whether neural networks are able to build an abstract sentence representation rather than capture surface statistical regularities. |
| Outcome: | The proposed model can achieve high accuracy on the long-range French object-verb agreement, indicating a possible flaw in the model's syntactic ability. |
Serial Position Effects of Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Serial position effects (SPE) are well-documented cognitive biases in human behavior. |
| Approach: | They propose to use binary choices instead of multiple choices where feasible . they also suggest limiting prompt length and placing crucial information at the beginning of prompts . |
| Outcome: | The proposed framework shows that the effects are widespread across LLMs and the proposed mitigation methods are effective. |
Missing the Margins: A Systematic Literature Review on the Demographic Representativeness of LLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | 211 studies on the demographic representativeness of large language models have conflicting results . 29% of the studies report positive conclusions on the representativeness, 30% do not evaluate LLMs across multiple demographic categories or within demographic subcategories. |
| Approach: | 211 papers review the representativeness of large language models . authors recommend more precise evaluation methods and comprehensive documentation of demographic attributes . |
| Outcome: | 211 studies on the representativeness of large language models are reviewed . 29% of the studies report positive conclusions, but 30% fail to specify subcategories . authors recommend more precise evaluation methods and documentation of demographic attributes . |
The Functional Relevance of Probed Information: A Case Study (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that transformer models like BERT rely on number information encoded in their representations of sentences’ subjects and head verbs when performing subject-verb agreement. |
| Approach: | They propose to use probing to find out which words contain functionally relevant information encoded in the representations of subject plurality and words that agree with it in number in BERT. |
| Outcome: | The proposed model only uses the subject plurality information encoded in its representations of the subject and words that agree with it in number. |
Predicting Reference: What do Language Models Learn about Discourse Models? (2020.emnlp-main)
Copied to clipboard
| Challenge: | a growing literature that probes neural language models to assess their latent acquisition of grammatical knowledge has not investigated their acquisition of discourse modeling ability. |
| Approach: | They draw on a psycholinguistic literature that has established how different contexts affect referential biases concerning who is likely to be referred to next. |
| Outcome: | The proposed models do not resemble human language users, the authors show . their models capture the linguistic knowledge required to perform discourse modeling . |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |