Measuring and Narrowing the Compositionality Gap in Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a language model can correctly answer all sub-problems but not generate the overall solution. |
| Approach: | They propose a method that asks itself and then answers follow-up questions to narrow the compositionality gap by reasoning explicitly instead of implicitly. |
| Outcome: | The proposed method improves on chain of thought by asking itself and answering follow-up questions. |
Similar Papers
Exploring the Compositional Deficiency of Large Language Models in Mathematical Reasoning Through Trap Problems (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current LLMs lack systematic compositionality, and therefore cannot serve as reliable cognitive models. |
| Approach: | They propose to introduce logical traps into the original problems of MATH and GSM8K to investigate the compositionality of large language models in mathematical reasoning. |
| Outcome: | The proposed model can generate infinite combinations from finite learned components. |
On the Challenges of Evaluating Compositional Explanations in Multi-Hop Inference: Relevance, Completeness, and Expert Ratings (2021.emnlp-main)
Copied to clipboard
| Challenge: | a large corpus of domain-expert relevance ratings augments a corpus for compositional explanations . a writer's study shows that the evaluations of compositional inference models underestimate performance . |
| Approach: | They construct a corpus of 126k domain-expert relevance ratings that augment explanations to standardized science exam questions. |
| Outcome: | The results show that evaluations underestimate performance of compositional explanations . they show that models regularly discover and produce valid explanations that are different than gold explanations. |
Mind the Gap: How BabyLMs Learn Filler-Gap Dependencies (2025.emnlp-main)
Copied to clipboard
| Challenge: | a long-standing debate concerns whether the linguistic input children receive is sufficient to explain the grammatical knowledge they develop. |
| Approach: | They evaluate baby language models trained on child-oriented input from the BabyLM Challenge and two base models trained in 10M and 100M tokens. |
| Outcome: | The proposed models acquire filler-gap dependencies but fail to generalize or fully capture island constraints. |
Quantifying Compositionality of Classic and State-of-the-Art Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Static word embeddings make strong claims about compositionality, but the SOTA generative models go too far in the other direction. |
| Approach: | a new study evaluates the compositionality of word embeddings by canonical correlation analysis . strong compositional signals are observed in later training stages across data modalities . |
| Outcome: | a new evaluation of compositional models shows that they exploit access meanings when justified . strong compositional signals are observed in later training stages and in deeper layers of the transformer-based model before a decline at the top layer. |
Toward Compositional Behavior in Neural Models: A Survey of Current Views (2024.emnlp-main)
Copied to clipboard
| Challenge: | Compositionality is a core property of natural language, and it is regarded as a key goal for modern NLP systems. |
| Approach: | They propose a conceptual framework to address compositionality in NLP . they propose to use this framework to survey researchers active in this area . |
| Outcome: | The proposed framework finds consensus on key points and suggests that scale alone is unlikely to achieve the desired behavior. |
Re-Tuning: Overcoming the Compositionality Limits of Large Language Models with Recursive Tuning (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to solve compositional tasks are limited by complexity and complexity. |
| Approach: | They propose a method that tunes large language models to break down a problem into subproblems, solve those subproblem, and combine the results. |
| Outcome: | The proposed method significantly improves model performance on three representative compositional tasks: integer addition, dynamic programming, and parity. |
The Paradox of the Compositionality of Natural Language: A Neural Machine Translation Case Study (2022.acl-long)
Copied to clipboard
| Challenge: | Obtaining human-like performance in NLP is often argued to require compositional generalisation. |
| Approach: | They re-instantiate three compositionality tests from the literature and reformulate them for neural machine translation. |
| Outcome: | The proposed models are more compositional than models trained on more data, the authors show . they also show that some non-compositional behaviours are mistakes, whereas others reflect natural variation in data. |
Geometric Signatures of Compositionality Across a Language Model’s Lifetime (2025.acl-long)
Copied to clipboard
| Challenge: | linguistic compositionality allows atoms to locally combine to create global meaning . a rich array of meanings at the level of a phrase may be explained by simple rules of composition. |
| Approach: | They propose to relate the degree of compositionality in a dataset to the intrinsic dimension of its representations under an LM, a measure of feature complexity. |
| Outcome: | The proposed model is based on a geometric view of the compositionality of a dataset and the intrinsic dimension of its representations under an LM. |
Are representations built from the ground up? An empirical examination of local composition in language models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Compositionality is a hallmark of human language, but many phrases are non-compositional . a study by a team of researchers shows that LMs may not be able to distinguish between compositional and non-composable phrases. |
| Approach: | They propose to predict LM-internal representations of longer phrases given their constituents . they find that the representation of a parent phrase can be predicted with some accuracy . |
| Outcome: | The proposed model can predict a parent phrase with some accuracy given its children's transformations, but this is not the case. |
From A and B to A+B: Can Large Language Models Solve Compositional Math Problems? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies that create problem variants by adding perturbations to a single problem focus on the interaction between problems. |
| Approach: | They propose a pipeline with 98.2% accuracy to combine two original problems with a logical connection and to evaluate LLMs' generalization ability on the compositional problems. |
| Outcome: | The proposed pipeline can combine two original problems with a logical connection to get a new math problem and evaluate its compositional generalization on the compositional problems. |