Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov
| Challenge: | a novel linear characteristic exclusive to transformer decoders is revealed: embedding transformations between sequential layers exhibit almost perfect linearity. |
| Approach: | They propose a cosine-similarity-based regularization to reduce layer linearity in transformer decoders. |
| Outcome: | The proposed method improves performance metrics on Tiny Stories and SuperGLUE but also decreases the linearity of the models. |
Similar Papers
Reservoir Transformers (2021.acl-long)
Copied to clipboard
| Challenge: | Using random initialization, we show that some transformers obtain impressive performance even when some of the layers are frozen. |
| Approach: | They propose to freeze transformer layers and use them to improve performance . they find that the transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. |
| Outcome: | The proposed model improves on translation and language modelling tasks even when some layers are frozen. |
Jump to Conclusions: Short-Cutting Transformers with Linear Transformations (2024.lrec-main)
Copied to clipboard
| Challenge: | Transformer-based language models create hidden representations of inputs at every layer, but only use final-layer representations for prediction. |
| Approach: | They propose a method for casting hidden representations as final representations, bypassing transformer computation in-between. |
| Outcome: | The proposed method produces more accurate predictions from hidden layers across various model scales, architectures, and data distributions. |
Sneaking Syntax into Transformer Language Models with Tree Regularization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for incorporating syntactic inductive biases into transformers are limited . we introduce auxiliary loss function that converts bracketing decisions into differentiable orthogonality constraints on vector hidden states. |
| Approach: | They propose to introduce syntactic inductive biases into transformer circuits through a structured regularizer. |
| Outcome: | The proposed approach could unlock more robust and data-efficient learning in transformer language models . it integrates seamlessly with the standard LM objective, requiring no architectural changes. |
The Devil in Linear Transformer (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing linear transformers suffer from performance degradations on various tasks and corpus. |
| Approach: | They propose a new linear attention that replaces scaling with a normalization to stabilize gradients and confine attention to neighbouring tokens in early layers. |
| Outcome: | The proposed model outperforms vanilla transformers on the long-range arena benchmark while being significantly more space-time efficient. |
How to Dissect a Muppet: The Structure of Transformer Embedding Spaces (2022.tacl-1)
Copied to clipboard
| Challenge: | Pretrained embeddings based on the Transformer architecture have taken the NLP community by storm . a novel decomposition of Transformer output embeddables is demonstrated . |
| Approach: | They propose to decompose Transformer output embeddings into a sum of vector factors . they show multi-head attentions and feed-forwards are not equally useful in downstream applications . |
| Outcome: | The proposed method outperforms recurrent architectures on a wide variety of tasks. |
Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (2022.tacl-1)
Copied to clipboard
| Challenge: | a novel class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers and recursive syntactic compositions. |
| Approach: | They introduce Transformer Grammars, a class of Transformer language models that combine expressive power and recursive syntactic compositions. |
| Outcome: | The proposed model outperforms strong baselines on sentence-level language modeling perplexity and syntax-sensitive language evaluation metrics. |
Suppressing Final Layer Hidden State Jumps in Transformer Pretraining (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing models exhibit only slight changes in the angular distance between the input and output hidden state vectors in the middle layers . |
| Approach: | They propose a jump-suppressing regularizer which penalizes large hidden state displacements near the final layer during pre-training. |
| Outcome: | The proposed method significantly reduces hidden state jumps in the final layer and increases model capacity. |
Choose Your Transformer: Improved Transferability Estimation of Transformer Models on Classification Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for NLP tasks require fine-tuning, but it is computationally infeasible. |
| Approach: | They propose an approach that inexpensively estimates a ranking of the expected performance of a given set of transformer language models for a specific task. |
| Outcome: | The proposed model improves the Pearson correlation coefficient between the true model ranks and the estimate. |
Transformer-specific Interpretability (2024.eacl-tutorials)
Copied to clipboard
| Challenge: | Transformers are dominant play-ers in various scientific fields, but their inner workings remain opaque. |
| Approach: | This tutorial presents a trending approach to interpreting Transformers . it uses specific features of the Transformer architecture to quantify context- mixing interactions . |
| Outcome: | This tutorial aims to show how a new trending approach can be applied to Transformer-based models. |
Making Transformers Solve Compositional Tasks (2022.acl-long)
Copied to clipboard
| Challenge: | Several studies have reported the inability of Transformer models to generalize compositionally . a key aspect of natural language is the ability to learn basic primitives . |
| Approach: | They propose to use Transformers to generalize compositionally in a large range of tasks . they find that Transformers generalize significantly better than previous models . |
| Outcome: | The proposed models generalize compositionally significantly better than previous models . a set of 12 datasets shows that the proposed models can be improved . |