Sneaking Syntax into Transformer Language Models with Tree Regularization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for incorporating syntactic inductive biases into transformers are limited . we introduce auxiliary loss function that converts bracketing decisions into differentiable orthogonality constraints on vector hidden states. |
| Approach: | They propose to introduce syntactic inductive biases into transformer circuits through a structured regularizer. |
| Outcome: | The proposed approach could unlock more robust and data-efficient learning in transformer language models . it integrates seamlessly with the standard LM objective, requiring no architectural changes. |
Similar Papers
Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (2022.tacl-1)
Copied to clipboard
| Challenge: | a novel class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers and recursive syntactic compositions. |
| Approach: | They introduce Transformer Grammars, a class of Transformer language models that combine expressive power and recursive syntactic compositions. |
| Outcome: | The proposed model outperforms strong baselines on sentence-level language modeling perplexity and syntax-sensitive language evaluation metrics. |
Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervision (2024.findings-acl)
Copied to clipboard
| Challenge: | Syntactic Language Models (SLMs) have difficulty with inference efficiency due to explicit generation of syntactical structures. |
| Approach: | They propose a method to "plant" trees into attention weights of unidirectional Transformer LMs to implicitly reflect syntactic structures of natural language. |
| Outcome: | The proposed method outperforms SLMs on the SyntaxGym benchmark. |
Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers (2025.tacl-1)
Copied to clipboard
Kabir Ahuja, Vidhisha Balachandran, Madhur Panwar, Tianxing He, Noah A. Smith, Navin Goyal, Yulia Tsvetkov
| Challenge: | Inductive biases in transformers can cause hierarchical generalization without explicitly encoding structural bias. |
| Approach: | They investigate sources of inductive bias in transformer models and their training that could cause such preference for hierarchical generalization. |
| Outcome: | The proposed model can generalize to novel syntactic forms without explicit bias . the proposed model is able to generalize on a dataset with a hierarchical grammar . |
Do Transformers Parse while Predicting the Masked Word? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained language models encode linguistic structures like parse trees while being trained unsupervised. |
| Approach: | They propose to train pre-trained language models to encode linguistic structures like parse trees while unsupervised. |
| Outcome: | The proposed model performs optimally for masked language modeling loss on the English PCFG. |
Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees (2021.eacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models like BERT achieve superior performances in various NLP tasks without explicit consideration of syntactic information. |
| Approach: | They propose a plug-and-play framework that incorporates syntax trees into pre-trained Transformers. |
| Outcome: | The proposed framework improves on pre-trained models on natural language understanding datasets and shows that it can be used to train pre-structured neural networks. |
Rethinking Positional Encoding in Tree Transformer for Code Representation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent works have proposed novel tree Transformers to capture the syntactic structure in source code. |
| Approach: | They propose a novel tree Transformer encoding node positions based on a description method for tree structures to incorporate inductive bias into Transformer. |
| Outcome: | The proposed model outperforms baselines on code summarization and completion tasks across two languages, and it is able to perform better on both local and global paradigms. |
How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases (2023.acl-long)
Copied to clipboard
| Challenge: | a recent study found that pre-training can teach language models to rely on hierarchical syntactic features . aaron ramirez: we find that pretraining on simpler language induces a hierarchic bias . |
| Approach: | They find that pre-training can teach language models to rely on hierarchical syntactic features . authors: this suggests that in cognitively plausible language acquisition settings, models may be more data-efficient . |
| Outcome: | a recent study shows that pre-training can teach language models to rely on hierarchical features . the findings suggest that in plausible language acquisition settings, language models may be more data-efficient than previously thought . |
Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale (2024.acl-long)
Copied to clipboard
| Challenge: | Existing syntactic language models require a gold tree and sequential training to generate sentences. |
| Approach: | They propose an unsupervised syntactic language model that incrementally generates a sentence with its syntaktic tree in a left-to-right manner. |
| Outcome: | The proposed model outperforms existing models on grammar induction and comprehension tasks while holding a substantial acceleration on training. |
A Systematic Study of Compositional Syntactic Transformer Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Syntactic language models (SLMs) incorporate syntactical biases into Transformers . authors identify key aspects of design choices in existing models and novel variants based on experimental results . |
| Approach: | They propose a framework that incorporates existing and new SLMs to enhance Transformers by incorporating syntactic biases. |
| Outcome: | The proposed framework improves on existing models and novel variants across language modeling, syntactic generalization, summarization, and inference efficiency. |
Can Transformers Learn n-gram Language Models? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has tested transformers' ability to represent formal languages, but language models are not classifiers of strings but rather distributions over them. |
| Approach: | They relate transformers' ability to learn random n-gram language models to ngram language model (LM) they find add- smoothing outperforms transformers on the former, while transformers perform better on the latter . |
| Outcome: | The proposed models outperform classical methods designed to learn n-gram LMs, while transformers perform better on the latter. |