The Paradox of the Compositionality of Natural Language: A Neural Machine Translation Case Study (2022.acl-long)
Copied to clipboard
| Challenge: | Obtaining human-like performance in NLP is often argued to require compositional generalisation. |
| Approach: | They re-instantiate three compositionality tests from the literature and reformulate them for neural machine translation. |
| Outcome: | The proposed models are more compositional than models trained on more data, the authors show . they also show that some non-compositional behaviours are mistakes, whereas others reflect natural variation in data. |
Similar Papers
On Compositional Generalization of Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Modern neural machine translation models have shown competitive performance in benchmarks such as WMT, but there are significant issues such as robustness, domain generalization, etc. |
| Approach: | They propose a benchmark dataset for NMT models from the perspective of compositional generalization and quantitatively analyze the results. |
| Outcome: | The proposed model performs well under traditional metrics, but is low in out-of-domain and low-resource conditions. |
On Evaluating Multilingual Compositional Generalization with Translated Datasets (2023.acl-long)
Copied to clipboard
| Challenge: | a growing amount of research investigating compositional generalization in NLP is done on English . a critical semantic distortion is a limitation of the translation of datasets . |
| Approach: | They propose to translate a dataset for evaluating compositional generalization in semantic parsing. |
| Outcome: | The proposed benchmarks show that the translation of the MCWQ dataset suffers from semantic distortion. |
Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to semantic parsing only evaluated on synthetic datasets that are not representative of natural language variation. |
| Approach: | They propose a semantic parsing approach that handles both natural language variation and compositional generalization. |
| Outcome: | The proposed model outperforms existing models across compositional generalization challenges on non-synthetic datasets while being competitive with the state-of-the-art on standard evaluations. |
Compositional Generalization by Factorizing Alignment and Translation (2020.acl-srw)
Copied to clipboard
| Challenge: | a crucial property underlying the expressive power of human language is its systematicity. |
| Approach: | They propose to make an analogous separation between alignment and translation in neural machine translation to capture compositional structure. |
| Outcome: | The proposed architecture outperforms existing neural networks on a compositional generalization task without supervision. |
Defending Compositionality in Emergent Languages (2022.naacl-srw)
Copied to clipboard
| Challenge: | a recent paper has suggested that compositionality is a key factor in language productivity, but some research has questioned this. |
| Approach: | They argue that compositionality is essential for successful generalization . they run a two-agent communication game to test this hypothesis . |
| Outcome: | The proposed results show that ANNs can generalize well even without compositional behavior . authors argue that the results are incomplete and weak . |
Toward Compositional Behavior in Neural Models: A Survey of Current Views (2024.emnlp-main)
Copied to clipboard
| Challenge: | Compositionality is a core property of natural language, and it is regarded as a key goal for modern NLP systems. |
| Approach: | They propose a conceptual framework to address compositionality in NLP . they propose to use this framework to survey researchers active in this area . |
| Outcome: | The proposed framework finds consensus on key points and suggests that scale alone is unlikely to achieve the desired behavior. |
Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language? (2020.acl-main)
Copied to clipboard
| Challenge: | Despite the success of language models using neural networks, it remains unclear to what extent neural models have the generalization ability to perform inferences. |
| Approach: | They propose a method to evaluate whether neural models can learn systematicity of monotonicity inference in natural language. |
| Outcome: | The proposed method shows that neural models can perform inferences on unseen combinations of lexical and logical phenomena when syntactic structures are similar between training and test sets. |
Do Neural Language Models Inferentially Compose Concepts the Way Humans Can? (2024.lrec-main)
Copied to clipboard
| Challenge: | a new study shows that language models and humans may rely on different approaches to represent and compose lexical items across sentence structure. |
| Approach: | They propose to use a dataset to test the performance of neural language models and humans on inferentially driven conceptual compositions. |
| Outcome: | The proposed model elicits probability estimates for a noun in a minimally composed phrase . RoBERTa, BERT-large, and GPT-2 exhibited the closest resemblance to human responses . |
The Learnability of Model-Theoretic Interpretation Functions in Artificial Neural Networks (2026.findings-acl)
Copied to clipboard
| Challenge: | Entity vectors improve scores on basic event, while gated architectures benefit most. |
| Approach: | They extend entity-level semantic representations, modern architectures, principled competing event generation, extended systematicity tests and a two-dimensional difficulty analysis disaggregating results by modifier complexity. |
| Outcome: | The proposed model-theoretic interpretation functions generalize systematically to out-of-training-sample sentences. |
Evaluating Structural Generalization in Neural Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have focused on compositional generalization with semantic parsing, but it remains unclear to what extent models can translate sentences that require structural generalization. |
| Approach: | They construct a machine translation dataset that measures compositional generalization with control of words and sentence structures. |
| Outcome: | The proposed model struggle more in structural generalization than in compositional generalization. |