Fine-Tuning Pre-trained Transformers into Decaying Fast Weights (2022.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive Transformers incur O(T) complexity during per-token generation due to the self-attention mechanism. |
| Approach: | They propose a kernel-based method to approximate causal self-attention by replacing it with recurrent formulations with various update rules and feature maps to achieve O(1) time and memory complexity. |
| Outcome: | The proposed method outperforms prior methods and retains 99% of attention’s performance on WikiText-103 against more complex attention substitutes. |
Similar Papers
Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies of the computational power of recurrent neural networks reveal a hierarchy of RNN architectures, given finite-precision assumptions. |
| Approach: | They propose to use auto-regressive Transformers with linearised attention to build RNNs . they show that many well-known results for the standard Transformer directly transfer to LTs - a new approach is proposed . |
| Outcome: | The proposed extensions overcome limitations of the LT and self-referential weight matrices. |
Token-Wise Kernels (TWiKers) for Vicinity-Aware Attention in Transformers (2026.findings-eacl)
Copied to clipboard
| Challenge: | Token-Wise Kernels (TWiKers) are a novel enhancement to transformers that learn token-specific convolutional kernels applied to the keys or values. |
| Approach: | They propose a transformer enhancement that learns token-specific convolutional kernels applied to the keys or values. |
| Outcome: | The proposed transformers learn token-specific convolutional kernels applied to the keys or values . the results show that content words retain self-focus while function words shift attention toward their neighbors . |
Finetuning Pretrained Transformers into RNNs (2021.emnlp-main)
Copied to clipboard
Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, Noah A. Smith
| Challenge: | Efficient transformers outperform recurrent neural networks in natural language generation, but this comes with significant computational cost and memory footprint during generation. |
| Approach: | They propose to convert a pretrained transformer into its efficient recurrent counterpart, improving efficiency while maintaining accuracy. |
| Outcome: | The proposed transformers outperform recurrent neural networks in natural language generation but come with significant computational and memory footprint during generation. |
Recurrent Attention for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent research questions the importance of dot-product self-attention in Transformer models and shows that most attention heads learn simple positional patterns. |
| Approach: | They propose a novel mechanism to replace dot-product self-attention with a recurrent atteNtion mechanism that directly learns attention weights without token-to-token interaction. |
| Outcome: | The proposed model outperforms the Transformer model on translation tasks with fewer parameters and inference time. |
Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained transformers suffer from a computationally expensive self-attention mechanism that interacts with all tokens, including those unfavorable to classification performance. |
| Approach: | They propose to integrate token pruning and token combining strategies to improve model performance and reduce computational demands. |
| Outcome: | Experiments with various datasets show that the proposed model performs better than baseline models, with the best improvement over the existing model. |
Guiding Attention for Self-Supervised Learning with Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that self-attention patterns in trained models contain a majority of non-linguistic regularities. |
| Approach: | They propose a technique to allow efficient self-supervised learning with bi-directional Transformers by using an auxiliary loss function to guide attention heads to conform to such patterns. |
| Outcome: | The proposed method achieves state-of-the-art in low-resource settings and is agnostic to pre-training objectives. |
Chain and Causal Attention for Efficient Entity Tracking (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to handle entity tracking require at least log2 (n+1) layers to handle n state changes. |
| Approach: | They propose an efficient enhancement to the standard attention mechanism to handle long-term dependencies with a single layer. |
| Outcome: | The proposed model can handle entity tracking with n state changes with a single layer. |
Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a large computational cost for attention computation in large language models is a major obstacle . |
| Approach: | They propose a convolution-like structure for attention computation using convolution matrices . they then propose an efficient approximation method to approximate the attention matrix . |
| Outcome: | The proposed method achieves nearly linear time complexity in n1+o(1) time. |
On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers (2021.findings-acl)
Copied to clipboard
Tianchu Ji, Shraddhan Jain, Michael Ferdman, Peter Milder, H. Andrew Schwartz, Niranjan Balasubramanian
| Challenge: | Recent work shows that attention can be pruned to zeros with minimal loss in accuracy. |
| Approach: | They propose a pruning technique which quantizes attention to a 3-bit format without retraining . they find that 80% of attention values can be pruned to zeros with minimal loss in accuracy . |
| Outcome: | The proposed approach produces only a few unique attention values with minimal loss in accuracy. |
Integral Transformer: Denoising Attention, Not Too Much Not Too Little (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to reduce attention noise by integrating signals from logit distributions are prone to attention noise. |
| Approach: | They propose a self-attention mechanism that integrates signals from the logit distribution to denoise attention. |
| Outcome: | The proposed model outperforms vanilla, Cog, and Differential attention variants on knowledge and reasoning benchmarks. |