| Challenge: | Compositionality is the ability to combine familiar units like words into novel phrases and sentences. |
| Approach: | They introduce a set of dependency parses for Compositional Freebase Queries (CFQ) they analyze the behaviour of a state-of-the-art dependency parser on the CFQ dataset . |
| Outcome: | The proposed dependency parser performs lower on the most challenging splits with the highest compound divergence. |
Similar Papers
Grammar-based Decoding for Improved Compositional Generalization in Semantic Parsing (2023.findings-acl)
Copied to clipboard
| Challenge: | Sequence-to-sequence (seq2sequ) models have been successful in semantic parsing tasks but struggle on out-of-distribution data. |
| Approach: | They propose to use a large-scale dialogue dataset to evaluate compositional generalization of semantic parsing. |
| Outcome: | The proposed model outperforms BART- and T5-based models on the SMCalflow-CS dataset on the zero-shot learning task. |
Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to semantic parsing only evaluated on synthetic datasets that are not representative of natural language variation. |
| Approach: | They propose a semantic parsing approach that handles both natural language variation and compositional generalization. |
| Outcome: | The proposed model outperforms existing models across compositional generalization challenges on non-synthetic datasets while being competitive with the state-of-the-art on standard evaluations. |
Revisiting the Compositional Generalization Abilities of Neural Sequence Models (2022.acl-short)
Copied to clipboard
| Challenge: | Existing studies have suggested that standard seq-to-seq models lack the ability to generalize compositionally. |
| Approach: | They propose to use one-shot primitive generalization as introduced by the popular SCAN benchmark to modify the training distribution in simple and intuitive ways to achieve near-perfect generalization performance. |
| Outcome: | The proposed model achieves near-perfect generalization performance despite a lack of training data . |
Data Factors for Better Compositional Generalization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent diagnostic datasets on compositional generalization expose severe problems . state-of-the-art models trained on larger and more general datasets show better generalization ability . |
| Approach: | They conduct an empirical analysis by training Transformer models on a variety of training sets with different data factors including dataset scale, pattern complexity, example difficulty, etc. |
| Outcome: | The proposed model training on larger datasets improves on compositional generalization tasks. |
Improving Compositional Generalization in Classification Tasks via Structure Annotations (2021.acl-short)
Copied to clipboard
| Challenge: | Compositional generalization is the ability to generalize systematically to a new data distribution by combining known components. |
| Approach: | They propose to convert a natural language sequence-to-sequence dataset into a classification dataset that requires compositional generalization. |
| Outcome: | The proposed model can generalize compositionally by providing hints on the structure of the input. |
Evaluating Structural Generalization in Neural Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have focused on compositional generalization with semantic parsing, but it remains unclear to what extent models can translate sentences that require structural generalization. |
| Approach: | They construct a machine translation dataset that measures compositional generalization with control of words and sentence structures. |
| Outcome: | The proposed model struggle more in structural generalization than in compositional generalization. |
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022.emnlp-main)
Copied to clipboard
Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, Kristina Toutanova
| Challenge: | Pre-trained language models struggle on out-of-distribution compositional generalization . recent work shows considerable improvements on many NLP tasks from model scaling . |
| Approach: | They evaluate encoder-decoder models up to 11B parameters and decoder-only models up 540B parameters . they compare scaling curves for fine-tuning, prompt tuning, and in-context learning methods . |
| Outcome: | The proposed scaling methods improve compositional generalization on many tasks . fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional . larger models are better at modeling the syntax of the output space, the study finds . |
On Evaluating Multilingual Compositional Generalization with Translated Datasets (2023.acl-long)
Copied to clipboard
| Challenge: | a growing amount of research investigating compositional generalization in NLP is done on English . a critical semantic distortion is a limitation of the translation of datasets . |
| Approach: | They propose to translate a dataset for evaluating compositional generalization in semantic parsing. |
| Outcome: | The proposed benchmarks show that the translation of the MCWQ dataset suffers from semantic distortion. |
SLOG: A Structural Generalization Benchmark for Semantic Parsing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing compositional generalization benchmarks focus on lexical generalisation, the interpretation of novel lexicals in syntactic structures familiar from training. |
| Approach: | They propose a semantic parsing dataset that extends COGS with 17 structural generalization cases to evaluate how well models generalize to new complex linguistic expressions. |
| Outcome: | The proposed model generalization accuracy is far below the near-perfect accuracy of existing models on COGS, demonstrating the role of SLOG in foregrounding the large discrepancy between models’ lexical and structural generalization capacities. |
Improving Compositional Generalization in Semantic Parsing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Generalization of models to out-of-distribution data has sparked substantial interest . compositional generalization is the ability to systematically generalize to test examples composed of components seen during training . |
| Approach: | They propose to extend compositional generalization in semantic parsing by using contextual representations and training attention to agree with pre-computed token alignments. |
| Outcome: | The proposed extensions improve compositional generalization on OOD compositions. |