When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent benchmarks like ReaSCAN use navigation tasks grounded in a grid world to assess whether neural models exhibit compositional behaviour. |
| Approach: | They propose a transformer-based model that outperforms specialized architectures on ReaSCAN and a modified version of gSCAN to test their performance. |
| Outcome: | The proposed model outperforms specialized architectures on ReaSCAN and gSCAN on a grid world and can generalize to deeper input structures. |
Similar Papers
Making Transformers Solve Compositional Tasks (2022.acl-long)
Copied to clipboard
| Challenge: | Several studies have reported the inability of Transformer models to generalize compositionally . a key aspect of natural language is the ability to learn basic primitives . |
| Approach: | They propose to use Transformers to generalize compositionally in a large range of tasks . they find that Transformers generalize significantly better than previous models . |
| Outcome: | The proposed models generalize compositionally significantly better than previous models . a set of 12 datasets shows that the proposed models can be improved . |
Analyzing the Inner Workings of Transformers in Compositional Generalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on compositional generalization abilities of neural models have focused on benchmarks, but the results do not reflect the underlying competence of the model. |
| Approach: | They propose to find an existing subnetwork that contributes to the generalization performance and perform causal analyses on how the model utilizes syntactic features. |
| Outcome: | The proposed model relies on syntactic features but the subnetwork with better generalization performance relies mainly on a non-compositional algorithm . |
The Impact of Depth on Compositional Generalization in Transformer Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | In this paper, we test the hypothesis that deeper transformers generalize more compositionally. |
| Approach: | They propose to add layers to transformers to generalize more compositionally . they propose to fine-tune the models so that the total number of parameters is constant . |
| Outcome: | The proposed model generalizes more compositionally than shallower models, but returns diminish . the proposed model can be made shallower without sacrificing performance . |
Compositional Networks Enable Systematic Generalization for Grounded Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that deep networks can mimic some human language abilities when presented with novel sentences . a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains is critical to building safe and fair robots, says a new study. |
| Approach: | They build a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains. |
| Outcome: | a new network generalizes its language understanding to compositional domains while generalizing its knowledge when prior work does not. |
The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that basic configurations can improve the performance of neural networks on systematic generalization. |
| Approach: | They propose to revisit basic configurations to improve the performance of Transformers on systematic generalization by revisiting scaling of embeddings, early stopping, relative positional embeddment, and Universal Transformer variants. |
| Outcome: | The proposed models improve accuracy from 50% to 85% on the PCFG productivity split and from 35% to 81% on COGS. |
Inducing Transformer’s Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing neural models lack systematic compositionality in learning symbolic structures . existing models lack this ability in learning symbols, despite being able to understand complex structures. |
| Approach: | They propose to use auxiliary sequence prediction tasks to train a Transformer model to understand compositional symbolic structures of input data. |
| Outcome: | The proposed model improves on the SCAN compositionality challenge, with only 418 (5%) training instances, and achieves 97.8% accuracy on the MCD1 split. |
Systematic Generalization on gSCAN: What is Nearly Solved and What is Next? (2021.emnlp-main)
Copied to clipboard
| Challenge: | a general-purpose Transformer-based model with crossmodal attention solves most of the systematic generalization problems . current models are data inefficient given the narrow scope of commands in gSCAN . |
| Approach: | They propose to use a Transformer-based model with cross-modal attention to solve gSCAN . they propose to generate data to incorporate relations between objects in the visual environment . |
| Outcome: | The proposed model outperforms specialized approaches on most splits, and is data inefficient given the narrow scope of commands. |
Compositional Generalization in Grounded Language Learning via Induced Model Sparsity (2022.naacl-srw)
Copied to clipboard
| Challenge: | induced model sparsity can help achieve compositional generalization and sample efficiency in grounded language learning problems. |
| Approach: | They propose a model that encourages sparse correlations between words and attributes to find a goal in a language-conditioned navigation problem with disentangled observations. |
| Outcome: | The proposed agent maintains high performance even when learning from a handful of demonstrations. |
Unravelling the Logic: Investigating the Generalisation of Transformers in Numerical Satisfiability Problems (2025.acl-long)
Copied to clipboard
| Challenge: | Transformer models exhibit minimal scale and noise invariance, along with limited vocabulary and number invariancy. |
| Approach: | They probe the generalisation prowess of Transformer models with respect to the hitherto unexplored domain of numerical satisfiability problems. |
| Outcome: | The proposed models exhibit minimal scale and noise invariance, along with limited vocabulary and number invariancy. |
Harnessing Dataset Cartography for Improved Compositional Generalization in Transformers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to understanding compositional generalization of models have focused on novel architectures and alternative learning paradigms. |
| Approach: | They propose a method that harnesses the power of dataset cartography to improve model accuracy by strategically identifying a subset of compositional generalization data. |
| Outcome: | The proposed method improves model accuracy by 10% on CFQ and COGS datasets. |