| Challenge: | Several efficient transformers have been proposed, but they all have a finite memory capacity and are forced to drop old information. |
| Approach: | They propose an unbounded long-term memory extension that extends the vanilla transformer by using a continuous-space attention mechanism to attend over the long-time memory. |
| Outcome: | The proposed model can model arbitrarily long contexts while keeping the computation budget fixed. |
Similar Papers
Adaptive Attention Span in Transformers (P19-1)
Copied to clipboard
| Challenge: | We extend the maximum context size of a neural network called Transformer to 8k characters. |
| Approach: | They propose a self-attention mechanism that can learn its optimal attention span . this allows for models with longer context and the capability to catch longer dependencies. |
| Outcome: | The proposed model achieves state-of-the-art performance on text8 and enwiki8 using 8k characters with no loss of performance, and maintains control over memory footprint and computational time. |
Do Transformers Need Deep Long-Range Memory? (2020.acl-main)
Copied to clipboard
| Challenge: | Deep attention models have advanced the modelling of sequential data across many domains. |
| Approach: | They propose to use a Transformer augmented with a long-range memory to model sequential data across many domains. |
| Outcome: | The Transformer-XL has a long-range memory at every layer of the network, rendering its state thousands of times larger than RNN predecessors. |
The NLP Task Effectiveness of Long-Range Transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing benchmarks on long-range attention models have not been sufficient to develop efficient Transformers and their practical application on complex NLP tasks. |
| Approach: | They propose to benchmark 7 Transformer variants on 5 difficult NLP tasks and 7 datasets to examine their capacity for long-range attention. |
| Outcome: | The proposed models have advantages on content selection and query-guided decoding, but they come with previously unrecognized drawbacks such as insufficient attention to distant tokens and accumulated approximation error. |
EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devices (2025.acl-industry)
Copied to clipboard
| Challenge: | Existing KV cache optimizations struggle with irreversible token eviction in long-output tasks . alternative sequence modeling architectures prove costly to adopt within established Transformer infrastructures. |
| Approach: | They propose a memory-efficient solution for infinite contexts that integrates compressed memory into Transformer-based LLMs through a trainable memory-gating module. |
| Outcome: | The proposed solution achieves comparable performance to baseline Transformer-based LLMs while optimizing memory consumption and time to first token. |
LongT5: Efficient Text-To-Text Transformer for Long Sequences (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work has shown that increasing the input length or increasing model size can improve the performance of Transformer-based neural models. |
| Approach: | They propose a model that integrates attention ideas from long-input transformers and adopts pre-training strategies from summarization pre-train into the scalable T5 architecture. |
| Outcome: | The proposed model outperforms the original T5 models on several summarization and question answering tasks and achieves state-of-the-art results. |
Adaptive Attention for Sparse-based Long-sequence Transformer (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that Transformers can process longer sequences because of their complexity and time scales quadratic to the sequence length. |
| Approach: | They propose an efficient Transformer model with adaptive attention that can select useful tokens automatically in sparse attention by learnable position vectors. |
| Outcome: | The proposed model can select useful tokens automatically in sparse attention by learnable position vectors. |
Octopus: Gated Selective Attention for Memory-Bounded Long-Context Inference in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Subquadratic architectures rely on aggressive state compression that degrades performance on complex reasoning tasks. |
| Approach: | They propose a framework that confers fixed-memory inference onto pretrained Transformers . they use a learnable module that enforces an adaptive sparsity policy over the context history . |
| Outcome: | The proposed framework outperforms state-of-the-art linearized baselines on the GSM8K benchmark by over 36 points under identical memory constraints. |
Continuous-Time Attention: PDE-Guided Mechanisms for Long-Sequence Transformers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to optimize attention for long sequences have been limited by their computational cost. |
| Approach: | They propose a framework that infuses partial differential equations into the Transformer’s attention mechanism to better handle long sequences. |
| Outcome: | The proposed framework achieves consistent performance gains over standard and long-sequence Transformer variants across a range of tasks. |
Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for length extrapolation are tailored for natural language modeling, a task known to have strong recency bias. |
| Approach: | They propose two attention alignment strategies to improve T5's long-context utilization capability without fine-tuning. |
| Outcome: | The proposed methods improve the long-context utilization capability of T5 on language modeling, retrieval, multi-document question answering, and code completion tasks without any fine-tuning. |
Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained transformer models have demonstrated remarkable performance across various natural language processing tasks. |
| Approach: | They propose a transformer variant with mixed attention spans that leverages the attention mechanism to capture long- and short-range dependencies in the sequence. |
| Outcome: | The proposed model can achieve competitive performance to models with full attention while reducing computational cost (75%) |