| Challenge: | Existing approaches to capture global context dependencies in sequence modeling suffer from quadratic complexity in time and memory usage. |
| Approach: | They propose an efficient Transformer architecture for fast long-range sequence modeling with a sparse attention matrix and a hidden state cross module. |
| Outcome: | The proposed architecture outperforms the standard multi-head attention and its variants in various long-sequence tasks with low computational costs. |
Similar Papers
Adaptive Attention for Sparse-based Long-sequence Transformer (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that Transformers can process longer sequences because of their complexity and time scales quadratic to the sequence length. |
| Approach: | They propose an efficient Transformer model with adaptive attention that can select useful tokens automatically in sparse attention by learnable position vectors. |
| Outcome: | The proposed model can select useful tokens automatically in sparse attention by learnable position vectors. |
Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing transformer models are computationally demanding and prohibitively costly for long sequences due to the quadratic complexity of its selfattention module. |
| Approach: | They propose a transformer-based model that inherits weights from large pretrained models by removing redundancies in hidden sequences using the ready-made Fast Fourier Transform operator. |
| Outcome: | The proposed model outperforms the standard BART model on the long-range modeling benchmark LRA with significant improvements in speed and space. |
Efficient Content-Based Sparse Attention with Routing Transformers (2021.tacl-1)
Copied to clipboard
| Challenge: | Self-attention suffers from quadratic computation and memory requirements with respect to sequence length . despite its effectiveness, self-attention models suffer from quadratic computation and a limited set of locations . |
| Approach: | They propose to learn dynamic sparse attention patterns that avoid allocating computation and memory to attend to content unrelated to the query of interest. |
| Outcome: | The proposed model outperforms similar sparse attention models on language modeling and image generation on Wikitext-103 . |
Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained transformer models have demonstrated remarkable performance across various natural language processing tasks. |
| Approach: | They propose a transformer variant with mixed attention spans that leverages the attention mechanism to capture long- and short-range dependencies in the sequence. |
| Outcome: | The proposed model can achieve competitive performance to models with full attention while reducing computational cost (75%) |
Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a large computational cost for attention computation in large language models is a major obstacle . |
| Approach: | They propose a convolution-like structure for attention computation using convolution matrices . they then propose an efficient approximation method to approximate the attention matrix . |
| Outcome: | The proposed method achieves nearly linear time complexity in n1+o(1) time. |
Long Sequence Modeling with Attention Tensorization: From Sequence to Tensor Learning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a lack of attention-based models for long sequences poses challenges for long-sequence modeling . attention tensorization can be used to extend context lengths with improved efficiency . tenorization enables training of LLMs with context length longer than those trained on . |
| Approach: | They propose to tensorize long input sequences into compact tenses followed by attention on each transformed dimension. |
| Outcome: | The proposed model can be used as efficient transformer backbones to extend input context length with improved memory and time efficiency. |
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Sparse attention is a promising strategy to extend long-context capabilities in LLMs . but its efficiency–accuracy trade-offs remain unclear due to the lack of comprehensive evaluation . |
| Approach: | They evaluate sparse attention methods across multiple model families and sizes . they find larger sparser models outperform smaller dense ones at equivalent cost . |
| Outcome: | The proposed methods outperform smaller sparse models at equivalent cost and improve the Pareto frontier. |
Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing models for long sequences are not efficient due to the quadratic space and time complexity of the self-attention modules. |
| Approach: | They propose to reduce the quadratic complexity to linear (modulo logarithmic factors) by low-dimensional projection and row selection. |
| Outcome: | The proposed methods outperform transformer-based models with smaller time/space footprint on the Long Range Arena benchmark. |
Fine-Tuning Pre-trained Transformers into Decaying Fast Weights (2022.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive Transformers incur O(T) complexity during per-token generation due to the self-attention mechanism. |
| Approach: | They propose a kernel-based method to approximate causal self-attention by replacing it with recurrent formulations with various update rules and feature maps to achieve O(1) time and memory complexity. |
| Outcome: | The proposed method outperforms prior methods and retains 99% of attention’s performance on WikiText-103 against more complex attention substitutes. |
The NLP Task Effectiveness of Long-Range Transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing benchmarks on long-range attention models have not been sufficient to develop efficient Transformers and their practical application on complex NLP tasks. |
| Approach: | They propose to benchmark 7 Transformer variants on 5 difficult NLP tasks and 7 datasets to examine their capacity for long-range attention. |
| Outcome: | The proposed models have advantages on content selection and query-guided decoding, but they come with previously unrecognized drawbacks such as insufficient attention to distant tokens and accumulated approximation error. |