COGS: A Compositional Generalization Challenge Based on Semantic Interpretation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Natural language is characterized by compositionality: meaning of complex expressions is constructed from the meanings of its constituent parts. |
| Approach: | They propose a semantic parsing dataset based on a fragment of English to assess compositional generalization abilities. |
| Outcome: | The proposed model can generalize meanings in a given sentence in 96–99% of the tests, but generalization accuracy is lower and the generalization sensitivity is higher. |
Similar Papers
On Evaluating Multilingual Compositional Generalization with Translated Datasets (2023.acl-long)
Copied to clipboard
| Challenge: | a growing amount of research investigating compositional generalization in NLP is done on English . a critical semantic distortion is a limitation of the translation of datasets . |
| Approach: | They propose to translate a dataset for evaluating compositional generalization in semantic parsing. |
| Outcome: | The proposed benchmarks show that the translation of the MCWQ dataset suffers from semantic distortion. |
Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to semantic parsing only evaluated on synthetic datasets that are not representative of natural language variation. |
| Approach: | They propose a semantic parsing approach that handles both natural language variation and compositional generalization. |
| Outcome: | The proposed model outperforms existing models across compositional generalization challenges on non-synthetic datasets while being competitive with the state-of-the-art on standard evaluations. |
Compositional Generalization by Factorizing Alignment and Translation (2020.acl-srw)
Copied to clipboard
| Challenge: | a crucial property underlying the expressive power of human language is its systematicity. |
| Approach: | They propose to make an analogous separation between alignment and translation in neural machine translation to capture compositional structure. |
| Outcome: | The proposed architecture outperforms existing neural networks on a compositional generalization task without supervision. |
Improving Compositional Generalization in Classification Tasks via Structure Annotations (2021.acl-short)
Copied to clipboard
| Challenge: | Compositional generalization is the ability to generalize systematically to a new data distribution by combining known components. |
| Approach: | They propose to convert a natural language sequence-to-sequence dataset into a classification dataset that requires compositional generalization. |
| Outcome: | The proposed model can generalize compositionally by providing hints on the structure of the input. |
Evaluating Structural Generalization in Neural Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have focused on compositional generalization with semantic parsing, but it remains unclear to what extent models can translate sentences that require structural generalization. |
| Approach: | They construct a machine translation dataset that measures compositional generalization with control of words and sentence structures. |
| Outcome: | The proposed model struggle more in structural generalization than in compositional generalization. |
Combine to Describe: Evaluating Compositional Generalization in Image Captioning (2022.acl-srw)
Copied to clipboard
| Challenge: | Recent work on compositionality has focused on the ability to combine simpler concepts to understand & generate arbitrarily more complex conceptual structures. |
| Approach: | They propose to use a set of image captioning models to benchmark their compositional generalization properties. |
| Outcome: | The proposed models do not generalize in terms of systematicity and productivity, but are robust to synonym substitutions. |
Grammar-based Decoding for Improved Compositional Generalization in Semantic Parsing (2023.findings-acl)
Copied to clipboard
| Challenge: | Sequence-to-sequence (seq2sequ) models have been successful in semantic parsing tasks but struggle on out-of-distribution data. |
| Approach: | They propose to use a large-scale dialogue dataset to evaluate compositional generalization of semantic parsing. |
| Outcome: | The proposed model outperforms BART- and T5-based models on the SMCalflow-CS dataset on the zero-shot learning task. |
Improving Generalization in Language Model-based Text-to-SQL Semantic Parsing: Two Simple Semantic Boundary-based Techniques (2023.acl-short)
Copied to clipboard
| Challenge: | Pre-trained language models (LMs)2 have been adopted for semantic parsing due to their promising performance and straightforward architectures. |
| Approach: | They propose to use token preprocessing to preserve semantic boundaries of tokens produced by LM tokenizers and special tokens to mark the boundaries of aligned components. |
| Outcome: | The proposed techniques improve the performance of pre-trained language models on two text-to-SQL semantic parsing datasets. |
Evaluating Morphological Compositional Generalization in Large Language Models (2025.naacl-long)
Copied to clipboard
Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Duygu Ataman, Lonneke Van Der Plas
| Challenge: | Large language models (LLMs) have demonstrated significant progress in various natural language generation and understanding tasks. |
| Approach: | They define morphemes as compositional primitives and design a suite of generative and discriminative tasks to assess morphological productivity and systematicity. |
| Outcome: | The proposed models can identify individual morphological combinations better than chance, but their performance lacks systematicity, leading to significant accuracy gaps compared to humans. |
Improving Compositional Generalization in Semantic Parsing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Generalization of models to out-of-distribution data has sparked substantial interest . compositional generalization is the ability to systematically generalize to test examples composed of components seen during training . |
| Approach: | They propose to extend compositional generalization in semantic parsing by using contextual representations and training attention to agree with pre-computed token alignments. |
| Outcome: | The proposed extensions improve compositional generalization on OOD compositions. |