| Challenge: | a recent study has shown that neural sequence models like transformers can generalize hierarchically when training for extended periods. |
| Approach: | They show that transformers can learn to generalize hierarchically after long training periods . they call this phenomenon structural grokking, which exhibits inverted U-shaped scaling in model depth . |
| Outcome: | The proposed model generalizes better than both very deep and very shallow models on multiple datasets. |
Similar Papers
Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers (2025.tacl-1)
Copied to clipboard
Kabir Ahuja, Vidhisha Balachandran, Madhur Panwar, Tianxing He, Noah A. Smith, Navin Goyal, Yulia Tsvetkov
| Challenge: | Inductive biases in transformers can cause hierarchical generalization without explicitly encoding structural bias. |
| Approach: | They investigate sources of inductive bias in transformer models and their training that could cause such preference for hierarchical generalization. |
| Outcome: | The proposed model can generalize to novel syntactic forms without explicit bias . the proposed model is able to generalize on a dataset with a hierarchical grammar . |
Hierarchical Transformers Are More Efficient Language Models (2022.findings-naacl)
Copied to clipboard
Piotr Nawrot, Szymon Tworkowski, Michał Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, Henryk Michalewski
| Challenge: | Transformers are impressive but inefficient and costly, which limits their applications and accessibility. |
| Approach: | They first use different ways to downsample and upsamplify activations in Transformers to make them hierarchical. |
| Outcome: | The proposed model outperforms Transformers on the ImageNet32 and enwik8 benchmarks. |
The Impact of Depth on Compositional Generalization in Transformer Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | In this paper, we test the hypothesis that deeper transformers generalize more compositionally. |
| Approach: | They propose to add layers to transformers to generalize more compositionally . they propose to fine-tune the models so that the total number of parameters is constant . |
| Outcome: | The proposed model generalizes more compositionally than shallower models, but returns diminish . the proposed model can be made shallower without sacrificing performance . |
Semantic Training Signals Promote Hierarchical Syntactic Generalization in Transformers (2024.emnlp-main)
Copied to clipboard
| Challenge: | Neural networks without hierarchical biases struggle to learn linguistic rules that come naturally to humans . et al., 2018: Transformers trained on form and meaning favor hierarchically generalization more than those trained on forms alone. |
| Approach: | They examine whether neural networks without hierarchical biases can generalize more like humans . they find that Transformers trained on form and meaning favor hierarchic generalization . |
| Outcome: | The proposed neural networks perform better on syntactic evaluations when trained on form and meaning compared to those trained on forms alone. |
Analyzing the Inner Workings of Transformers in Compositional Generalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on compositional generalization abilities of neural models have focused on benchmarks, but the results do not reflect the underlying competence of the model. |
| Approach: | They propose to find an existing subnetwork that contributes to the generalization performance and perform causal analyses on how the model utilizes syntactic features. |
| Outcome: | The proposed model relies on syntactic features but the subnetwork with better generalization performance relies mainly on a non-compositional algorithm . |
How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases (2023.acl-long)
Copied to clipboard
| Challenge: | a recent study found that pre-training can teach language models to rely on hierarchical syntactic features . aaron ramirez: we find that pretraining on simpler language induces a hierarchic bias . |
| Approach: | They find that pre-training can teach language models to rely on hierarchical syntactic features . authors: this suggests that in cognitively plausible language acquisition settings, models may be more data-efficient . |
| Outcome: | a recent study shows that pre-training can teach language models to rely on hierarchical features . the findings suggest that in plausible language acquisition settings, language models may be more data-efficient than previously thought . |
Is Grokking Worthwhile? Functional Analysis and Transferability of Generalization Circuits in Transformers (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models struggle with the "curse of two-hop reasoning" in compositional tasks. |
| Approach: | They propose to form a "Generalization Circuit" during a prolonged "grokking" phase . they argue that grokkking is the process of integrating memorized atomic facts into an easy-acquire reasoning path. |
| Outcome: | The proposed model is superior to non-grokked models, but it requires a large computational cost . the study shows that grokking is not the sudden acquisition of a new reasoning paradigm . |
How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech (2023.acl-long)
Copied to clipboard
| Challenge: | LSTMs and Transformers perform well at capturing the surface statistics of child-directed speech, but both model types generalize in a way consistent with an incorrect linear rule than the correct hierarchical rule. |
| Approach: | They train LSTMs and Transformers on text from the CHILDES corpus and evaluate what they learn about English yes/no questions. |
| Outcome: | The proposed models perform well at capturing the surface statistics of child-directed speech, but generalize more consistent with an incorrect linear rule than the correct hierarchical rule. |
Structural Guidance for Transformer Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained Transformer language models have proven remarkably successful in learning generic transferable linguistic representations without resorting to data intensive pre-training. |
| Approach: | They propose to combine a generative parsing and a structural scaffolding idea to guide the model's representation via additional structure loss that separates the incremental constituency parse. |
| Outcome: | The proposed models achieve impressive perplexity results on language modelling datasets, perform well on grammatical judgments, and provide useful linguistic representations that benefit a wide range of downstream tasks. |
Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that Transformers can recognize hierarchical structures without a specific positional encoding. |
| Approach: | They show that Transformers can generate hierarchical languages without a positional encoding . they also suggest that explicit positional encoders might have a detrimental effect on generalization . |
| Outcome: | The proposed model can generate hierarchical languages with respect to model size without encoding . |