Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Whether language models have inductive biases favoring typologically frequent grammatical properties over rare, implausible ones has been investigated, typically using artificial languages (ALs). |
| Approach: | They extend their context-free AL formalization by adopting Generalized Categorial Grammar (GCG) . they also examine the generalization ability of LMs to process unseen longer test sentences . |
| Outcome: | The proposed models better capture features of natural languages and can process unseen longer test sentences. |
Similar Papers
How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure (2023.tacl-1)
Copied to clipboard
| Challenge: | Competent speakers of a language know how likely a word w is to appear in a specific context . |
| Approach: | They use transformer-based large language models to generalize a novel noun argument . they show a bias to generalise based on linear order, instead of a linear order . |
| Outcome: | The proposed models perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training. |
Exploring Compositional Generalization of Large Language Models (2024.naacl-srw)
Copied to clipboard
| Challenge: | a recent study has found that large language models can generalize compositional instructions from simple instructions to complex ones. |
| Approach: | They study the generalization ability of large language models with respect to compositional instructions . they first construct a dataset with the help of ChatGPT guided by the self-instruct technique . |
| Outcome: | The proposed model can generalize from simple instructions to more intricate ones, the authors show . their results show that training LLMs on higher-order compositional instructions improves performance on lower-order ones, but not on higher order ones. |
Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacements (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks have exposed patterns and may not truly assess generalization ability of Large Language Models (LLMs). |
| Approach: | They propose a “Generalization Stress Test” to assess Large Language Models’ generalization ability under slight and controlled perturbations, including option length, problem types, and irrelevant noun replacements. |
| Outcome: | The proposed test shows that LLMs exhibit severe accuracy drops and unexpected biases when faced with minor but content-preserving modifications. |
Language Models as an Alternative Evaluator of Word Order Hypotheses: A Case Study in Japanese (2020.acl-main)
Copied to clipboard
| Challenge: | a method using neural language models (LMs) for analyzing the word order of language is currently lacking. |
| Approach: | They propose a method using neural language models to analyze the word order in Japanese . they test whether there is a parallel between LMs and human word order preference . |
| Outcome: | The proposed method is validated by comparing it with other linguistic studies. |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |
Can Language Models Learn Typologically Implausible Languages? (2026.tacl-1)
Copied to clipboard
| Challenge: | Language models provide a naturalistic framework for studying artificial language learning . authors: typological universals and tendencies are thought to be caused by a learning bias . |
| Approach: | They propose to train LMs on highly naturalistic counterfactual versions of English and Japanese . they show that LM learn subtly implausible languages more slowly . |
| Outcome: | The proposed language models learn subtly implausible languages more slowly compared to human models . the findings suggest that LMs exhibit typologically aligned learning preferences . |
Do Language Models Perform Generalizable Commonsense Inference? (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent work has applied pretrained language models to populate commonsense knowledge graphs (CKGs) but there is a lack of understanding on their generalization to multiple CKGs, unseen relations, and novel entities. |
| Approach: | They analyze the ability of pretrained language models to perform generalizable commonsense inference in terms of knowledge capacity, transferability and induction. |
| Outcome: | The proposed models can adapt to different schemas defined by multiple CKGs but fail to generalize to new relations. |
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks (2020.tacl-1)
Copied to clipboard
| Challenge: | Inductive biases can arise from any aspect of the model architecture, study finds . we investigate which architectural factors affect how models generalize . |
| Approach: | They investigate which architectural factors affect generalization behavior of neural network models . they use English question formation and English tense reinflection as test cases . |
| Outcome: | The findings suggest that human-like generalization requires architectural syntactic structure. |
Logic Haystacks: Probing LLMs’ Long-Context Logical Reasoning (Without Easily Identifiable Unrelated Padding) (2026.eacl-short)
Copied to clipboard
| Challenge: | Recent large language models claim long context windows, but evaluations often involve simple retrieval tasks or synthetic tasks padded with irrelevant text. |
| Approach: | They use grammars to generate simplified English with logical representations to create long input text while controlling its semantics. |
| Outcome: | The proposed model performs better with realistic distractors than with standard models. |
Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs (2025.acl-long)
Copied to clipboard
| Challenge: | LMs are highly flexible learners, capable of acquiring linguistic patterns beyond those learnable by humans. |
| Approach: | They train LMs to model impossible and typologically unattested languages . they find that the model does not achieve perfect separation between attested and unattest languages - suggesting some human-like inductive biases . |
| Outcome: | The proposed model can largely distinguish attested from impossible languages, but does not achieve perfect separation between them and their impossible counterparts. |