Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning? (2023.eacl-main)
Copied to clipboard
Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Ana Brassard, Masashi Yoshikawa, Keisuke Sakaguchi, Kentaro Inui
| Challenge: | Using a pre-trained dataset, we examine how well recent neural models capture compositionality in symbolic reasoning tasks. |
| Approach: | They propose a skill tree on compositionality that defines hierarchical levels of complexity along with three compositionality dimensions: systematicity, productivity, and substitutivity. |
| Outcome: | The proposed model struggled most with systematicity, performing poorly even with relatively simple compositions. |
Similar Papers
Defending Compositionality in Emergent Languages (2022.naacl-srw)
Copied to clipboard
| Challenge: | a recent paper has suggested that compositionality is a key factor in language productivity, but some research has questioned this. |
| Approach: | They argue that compositionality is essential for successful generalization . they run a two-agent communication game to test this hypothesis . |
| Outcome: | The proposed results show that ANNs can generalize well even without compositional behavior . authors argue that the results are incomplete and weak . |
The Paradox of the Compositionality of Natural Language: A Neural Machine Translation Case Study (2022.acl-long)
Copied to clipboard
| Challenge: | Obtaining human-like performance in NLP is often argued to require compositional generalisation. |
| Approach: | They re-instantiate three compositionality tests from the literature and reformulate them for neural machine translation. |
| Outcome: | The proposed models are more compositional than models trained on more data, the authors show . they also show that some non-compositional behaviours are mistakes, whereas others reflect natural variation in data. |
Neural-Symbolic Inference for Robust Autoregressive Graph Parsing via Compositional Uncertainty Quantification (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models excel at graph semantic parsing with rich annotated data, but generalize poorly to out-of-distribution and long-tail examples. |
| Approach: | They propose a compositionality-aware approach to neural-symbolic inference informed by model confidence to capture different aspects of the graph prediction. |
| Outcome: | The proposed method outperforms state-of-the-art models on an English resource grammar parsing problem on standard in-domain and seven OOD corpora. |
Measuring and Narrowing the Compositionality Gap in Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a language model can correctly answer all sub-problems but not generate the overall solution. |
| Approach: | They propose a method that asks itself and then answers follow-up questions to narrow the compositionality gap by reasoning explicitly instead of implicitly. |
| Outcome: | The proposed method improves on chain of thought by asking itself and answering follow-up questions. |
Quantifying Compositionality of Classic and State-of-the-Art Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Static word embeddings make strong claims about compositionality, but the SOTA generative models go too far in the other direction. |
| Approach: | a new study evaluates the compositionality of word embeddings by canonical correlation analysis . strong compositional signals are observed in later training stages across data modalities . |
| Outcome: | a new evaluation of compositional models shows that they exploit access meanings when justified . strong compositional signals are observed in later training stages and in deeper layers of the transformer-based model before a decline at the top layer. |
Testing the limits of logical reasoning in neural and hybrid models (2024.findings-naacl)
Copied to clipboard
| Challenge: | despite the successes of deep learning models, we still need to know more about how and what they learn. |
| Approach: | They create tests to analyze logical reasoning patterns in neural and hybrid models . they find that models can generalize logical thinking only to a limited degree . |
| Outcome: | The proposed models can capture elementary aspects of meaning but only to limited extent . authors say they need to understand how and what they learn . |
Laziness Is a Virtue When It Comes to Compositionality in Neural Semantic Parsing (2023.acl-long)
Copied to clipboard
Maxwell Crouse, Pavan Kapanipathi, Subhajit Chaudhury, Tahira Naseem, Ramon Fernandez Astudillo, Achille Fokoue, Tim Klinger
| Challenge: | Compositional generalization is a key feature of human intelligence and has been identified as a major point of weakness in neural methods for semantic parsing. |
| Approach: | They propose a neural parsing generation method that constructs logical forms from the bottom up, beginning from the logical form’s leaves. |
| Outcome: | The proposed method outperforms general-purpose parsers on a CFQ dataset and two other Text-to-SQL datasets while also being competitive with parser that have been tailored to each task. |
Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language? (2020.acl-main)
Copied to clipboard
| Challenge: | Despite the success of language models using neural networks, it remains unclear to what extent neural models have the generalization ability to perform inferences. |
| Approach: | They propose a method to evaluate whether neural models can learn systematicity of monotonicity inference in natural language. |
| Outcome: | The proposed method shows that neural models can perform inferences on unseen combinations of lexical and logical phenomena when syntactic structures are similar between training and test sets. |
Combine to Describe: Evaluating Compositional Generalization in Image Captioning (2022.acl-srw)
Copied to clipboard
| Challenge: | Recent work on compositionality has focused on the ability to combine simpler concepts to understand & generate arbitrarily more complex conceptual structures. |
| Approach: | They propose to use a set of image captioning models to benchmark their compositional generalization properties. |
| Outcome: | The proposed models do not generalize in terms of systematicity and productivity, but are robust to synonym substitutions. |
Recursive Neural Networks with Bottlenecks Diagnose (Non-)Compositionality (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Compositional generalisation is often investigated with artificial languages or highly-structured natural language data. |
| Approach: | They propose to use recursive neural models with bottlenecks to generalise compositionally for artificial languages. |
| Outcome: | The proposed model can generalise compositionally for natural language tasks without limiting the transfer of information between nodes. |