| Challenge: | Recent diagnostic datasets on compositional generalization expose severe problems . state-of-the-art models trained on larger and more general datasets show better generalization ability . |
| Approach: | They conduct an empirical analysis by training Transformer models on a variety of training sets with different data factors including dataset scale, pattern complexity, example difficulty, etc. |
| Outcome: | The proposed model training on larger datasets improves on compositional generalization tasks. |
Similar Papers
Improving Compositional Generalization in Classification Tasks via Structure Annotations (2021.acl-short)
Copied to clipboard
| Challenge: | Compositional generalization is the ability to generalize systematically to a new data distribution by combining known components. |
| Approach: | They propose to convert a natural language sequence-to-sequence dataset into a classification dataset that requires compositional generalization. |
| Outcome: | The proposed model can generalize compositionally by providing hints on the structure of the input. |
On Evaluating Multilingual Compositional Generalization with Translated Datasets (2023.acl-long)
Copied to clipboard
| Challenge: | a growing amount of research investigating compositional generalization in NLP is done on English . a critical semantic distortion is a limitation of the translation of datasets . |
| Approach: | They propose to translate a dataset for evaluating compositional generalization in semantic parsing. |
| Outcome: | The proposed benchmarks show that the translation of the MCWQ dataset suffers from semantic distortion. |
How Do In-Context Examples Affect Compositional Generalization? (2023.acl-long)
Copied to clipboard
| Challenge: | In-context learning paradigms that focus on large corpus are limiting compositional generalization performance. |
| Approach: | They propose a test suite to investigate in-context compositional generalization . they propose to use examples that are structurally similar to the test case . |
| Outcome: | The proposed test suite investigates in-context compositional generalization performance . it finds that the performance can be affected by the selection of in-const examples . |
Simple and effective data augmentation for compositional generalization (2024.naacl-long)
Copied to clipboard
| Challenge: | Compositional generalization is the ability of a system to correctly predict the meaning of complex sentences when trained on simpler sentences. |
| Approach: | They propose to use data augmentation methods to generate additional training data by sampling from an augmentation distribution to generalize to the out-of-distribution test data. |
| Outcome: | The proposed method outperforms existing methods that sampled from the training distribution and outperformed existing methods. |
The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that basic configurations can improve the performance of neural networks on systematic generalization. |
| Approach: | They propose to revisit basic configurations to improve the performance of Transformers on systematic generalization by revisiting scaling of embeddings, early stopping, relative positional embeddment, and Universal Transformer variants. |
| Outcome: | The proposed models improve accuracy from 50% to 85% on the PCFG productivity split and from 35% to 81% on COGS. |
Revisiting Generalization Across Difficulty Levels: It’s Not So Easy (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing research is mixed regarding whether training on easier or harder data leads to better results. |
| Approach: | They examine how well large language models generalize across different task difficulties by using a large dataset and a well-established difficulty metric. |
| Outcome: | The results show that training on hard data can't achieve consistent improvements across the full range of difficulties. |
Exploring Compositional Generalization of Large Language Models (2024.naacl-srw)
Copied to clipboard
| Challenge: | a recent study has found that large language models can generalize compositional instructions from simple instructions to complex ones. |
| Approach: | They study the generalization ability of large language models with respect to compositional instructions . they first construct a dataset with the help of ChatGPT guided by the self-instruct technique . |
| Outcome: | The proposed model can generalize from simple instructions to more intricate ones, the authors show . their results show that training LLMs on higher-order compositional instructions improves performance on lower-order ones, but not on higher order ones. |
Harnessing Dataset Cartography for Improved Compositional Generalization in Transformers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to understanding compositional generalization of models have focused on novel architectures and alternative learning paradigms. |
| Approach: | They propose a method that harnesses the power of dataset cartography to improve model accuracy by strategically identifying a subset of compositional generalization data. |
| Outcome: | The proposed method improves model accuracy by 10% on CFQ and COGS datasets. |
Analyzing the Inner Workings of Transformers in Compositional Generalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on compositional generalization abilities of neural models have focused on benchmarks, but the results do not reflect the underlying competence of the model. |
| Approach: | They propose to find an existing subnetwork that contributes to the generalization performance and perform causal analyses on how the model utilizes syntactic features. |
| Outcome: | The proposed model relies on syntactic features but the subnetwork with better generalization performance relies mainly on a non-compositional algorithm . |
Towards Understanding the Relationship between In-context Learning and Compositional Generalization (2024.lrec-main)
Copied to clipboard
| Challenge: | In-context learning is an inductive bias for compositional generalization, but many deep neural architectures struggle with this ability. |
| Approach: | They propose to force a causal Transformer to in-context learn to promote compositional generalization by using earlier examples to generalize to later ones. |
| Outcome: | The proposed model can solve 'ordinary' learning problems by utilizing earlier examples to generalize to later ones, i.e., in-context learning. |