Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacements (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks have exposed patterns and may not truly assess generalization ability of Large Language Models (LLMs). |
| Approach: | They propose a “Generalization Stress Test” to assess Large Language Models’ generalization ability under slight and controlled perturbations, including option length, problem types, and irrelevant noun replacements. |
| Outcome: | The proposed test shows that LLMs exhibit severe accuracy drops and unexpected biases when faced with minor but content-preserving modifications. |
Similar Papers
Unveiling the Generalization Power of Fine-Tuned Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional multitasking abilities, but the comprehensive effects of fine-tuning on the LLMs’ generalization ability are not fully understood. |
| Approach: | They conduct extensive experiments across five distinct language tasks on different datasets to investigate whether fine-tuning affects the generalization ability intrinsic to LLMs. |
| Outcome: | The proposed model can generalize to different domains and tasks by integrating the in-context learning strategy during fine-tuning on generation tasks. |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |
Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities in various tasks, but may rely on dataset biases as shortcuts for prediction. |
| Approach: | They propose to use a test suite to evaluate the impact of shortcuts on LLMs' performance. |
| Outcome: | The proposed test suite incorporates six shortcut types, five evaluation metrics, and four prompting strategies. |
Exploring Compositional Generalization of Large Language Models (2024.naacl-srw)
Copied to clipboard
| Challenge: | a recent study has found that large language models can generalize compositional instructions from simple instructions to complex ones. |
| Approach: | They study the generalization ability of large language models with respect to compositional instructions . they first construct a dataset with the help of ChatGPT guided by the self-instruct technique . |
| Outcome: | The proposed model can generalize from simple instructions to more intricate ones, the authors show . their results show that training LLMs on higher-order compositional instructions improves performance on lower-order ones, but not on higher order ones. |
Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Whether language models have inductive biases favoring typologically frequent grammatical properties over rare, implausible ones has been investigated, typically using artificial languages (ALs). |
| Approach: | They extend their context-free AL formalization by adopting Generalized Categorial Grammar (GCG) . they also examine the generalization ability of LMs to process unseen longer test sentences . |
| Outcome: | The proposed models better capture features of natural languages and can process unseen longer test sentences. |
Quantifying Generalizations: Exploring the Divide Between Human and LLMs’ Sensitivity to Quantification (2024.acl-long)
Copied to clipboard
| Challenge: | Generics are expressions used to communicate abstractions about categories . they allow for exceptions, and they are a powerful way to express knowledge about the world . |
| Approach: | They examine how large language models interpret generics to understand their meanings . they find that the presence of a generic sentence as context influences quantifiers based on the generalization . |
| Outcome: | The proposed models do not exhibit a strong sensitivity to quantification, the study finds . the results suggest that the presence of a generic sentence as context influences quantifiers . |
How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure (2023.tacl-1)
Copied to clipboard
| Challenge: | Competent speakers of a language know how likely a word w is to appear in a specific context . |
| Approach: | They use transformer-based large language models to generalize a novel noun argument . they show a bias to generalise based on linear order, instead of a linear order . |
| Outcome: | The proposed models perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training. |
Revisiting Generalization Across Difficulty Levels: It’s Not So Easy (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing research is mixed regarding whether training on easier or harder data leads to better results. |
| Approach: | They examine how well large language models generalize across different task difficulties by using a large dataset and a well-established difficulty metric. |
| Outcome: | The results show that training on hard data can't achieve consistent improvements across the full range of difficulties. |
Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent model editing techniques can introduce large unwanted side effects, a new study shows . existing specificity benchmarks do not detect these unwanted side-effects . a recent study shows that model edits can cause significant performance drop . |
| Approach: | They extend existing CounterFact benchmark to include a dynamic component and propose a new benchmark to evaluate model editing techniques. |
| Outcome: | The proposed benchmark improves existing benchmarks for specificity and avoids unwanted side effects. |
Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models have demonstrated impressive capabilities in various NLP tasks, but previous studies have shown they are sensitive to prompt wording and few-shot demonstrations and their order. |
| Approach: | They focus on LLMs robustness on multiple-choice questions . they find a performance gap of 13% to 85% when options are reordered . |
| Outcome: | The proposed model outperforms supervised models on multiple choice questions even outperforming humans. |