Squeezed Attention: Accelerating Long Context Length LLM Inference

Challenge:	Emerging Large Language Models require long input context to perform complex tasks.
Approach:	They propose an algorithm to reduce the complexity of attention with respect to the fixed context length.
Outcome:	The proposed method reduces the complexity of attention from linear to logarithmic with respect to the fixed context length.

Similar Papers

Overview

Challenge

Approach

Outcome

LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2024.acl-long)

Copied to clipboard

Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu

Semantic Text Processing

Challenge:	Longer prompts introduce irrelevant and redundant information, which can weaken LLMs' performance.
Approach:	They propose a prompt compression tool that improves LLMs' perception of key information in input prompts by up to 21.4% with around 4x fewer tokens in GPT-3.5-Turbo.
Outcome:	The proposed solution improves performance and reduces costs and latency by up to 21.4% with around 4x fewer tokens in the NaturalQuestions benchmark.

TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection (2025.emnlp-main)

Copied to clipboard

Wei Wu, Zhuoshi Pan, Kun Fu, Chao Wang, Liyi Chen, Yunchu Bai, Tianfu Wang, Zheng Wang, Hui Xiong

Responsible & Trustworthy NLP

Challenge:	Rapid advances in Large Language Models have spurred demand for processing extended context sequences . however, performance degradation due to sequence lengths out-of-distribution and excessively long inference times are limiting LLMs in long-context scenarios.
Approach:	They propose a training-free method for efficient and accurate long-context inference . they selectively involves a few critical KV cache tokens in attention calculation .
Outcome:	The proposed method speeds up attention computation and accelerates inference time while reducing selection overhead.

RefreshKV: Updating Small KV Cache During Long-form Generation (2025.acl-long)

Copied to clipboard

Fangyuan Xu, Tanya Goyal, Eunsol Choi

Challenge:	Existing methods for generating long sequences of tokens are expensive and require memory and computation resources.
Approach:	They propose a method that alternates between full context attention and attention over a subset of input tokens during generation.
Outcome:	The proposed method achieves comparable speedup to eviction-based methods while improving performance for various long-form generation tasks.

Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction (2026.findings-acl)

Copied to clipboard

Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, Shafiq Joty

Challenge:	Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long context inputs, but this comes at the cost of increased computational resources and latency.
Approach:	They propose an algorithm that uses early LLM layers as filters to select and compress input tokens, reducing the context length for subsequent processing.
Outcome:	The proposed method outperforms existing techniques on the Needle in a Haystack task while demonstrating comparable performance on the LongBench challenge.

APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUs (2025.acl-long)

Copied to clipboard

Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao, Weilin Zhao, Sun Ao, Hao Zhou, Jie Zhou, Zhiyuan Liu, Maosong Sun

Responsible & Trustworthy NLP

Challenge:	Long-context inference is crucial for advancing large language models, but its prefill speed remains a bottleneck.
Approach:	They propose an efficient long-context inference framework that leverages multi-host approximate attention to enhance prefill speed.
Outcome:	The proposed framework achieves speedups of 9.2, 4.2, and 1.6 without any degradation in performance.

LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification (2026.acl-long)

Copied to clipboard

Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du, Bo An

Green & Sustainable NLP

Challenge:	Large Language Models (LLMs) can process extremely long contexts, requiring efficient inference over extended inputs.
Approach:	They propose a model that uses a constant-sized key-value cache to train long-context models.
Outcome:	Experimental results show that LongSpec achieves 3.26x speedup over strong Flash Attention baselines and 2.34x wall clock time on four math reasoning tasks.

LLoCO: Learning Long Contexts Offline (2024.emnlp-main)

Copied to clipboard

Sijun Tan, Xiuyu Li, Shishir G Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph Gonzalez, Raluca Popa

Natural Language Interfaces

Challenge:	Large language models are still unable to handle long contexts due to the quadratic computational and memory overhead of the self-attention mechanism and the substantial KV cache sizes during generation.
Approach:	They propose a method to learn contexts offline through context compression and in-domain parameter-efficient finetuning with LoRA.
Outcome:	The proposed model outperforms in-context learning while using 30 fewer tokens during inference and significantly reduces the cost of long document question answering.

Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing (2026.acl-long)

Copied to clipboard

Lingkun Long, Yushi Huang, Shihao Bai, Ruihao Gong, Jun Zhang, Ao Zhou, Jianlei Yang

Green & Sustainable NLP

Challenge:	Existing methods for estimating attention importance for tokens are ineffective . dLLMs require bidirectional attention, which limits inference efficiency .
Approach:	They propose a training-free attention sparsification framework for efficient long-context inference . they propose 'sink-aware pruning strategy' to accurately estimate and remove redundant computation .
Outcome:	The proposed approach offers 29 lossless speedup under 32K context length.

From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models (2026.findings-acl)

Copied to clipboard

Chejian Xu, Wei Ping, Peng Xu, Zihan Liu, Boxin Wang, Mohammad Shoeybi, Bo Li, Bryan Catanzaro

Language Models

Challenge:	Long-context capabilities are essential for document and video understanding, in-contact learning, and inference-time scaling.
Approach:	They propose an efficient training recipe for building ultra-long context LLMs from aligned instruct model, pushing the boundaries of context lengths from 128K to 1M, 2M, and 4M tokens.
Outcome:	The proposed model extends the context window while maintaining short context capabilities while maintaining the performance of the existing model.

LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing (2026.findings-acl)

Copied to clipboard

Dongfang Li, Zixuan Liu, Gang Lin, Baotian Hu, Min Zhang

Green & Sustainable NLP

Challenge:	Existing retrieval-based methods compromise semantic integrity through fixed-size chunking and suffer from inefficient linear scanning.
Approach:	They propose a method that preserves local semantic coherence through boundary-aware chunking and constructs a recursive hierarchical index rooted in the triangle inequality.
Outcome:	The proposed method achieves 3.6 end-to-end inference speedup with negligible degradation in model performance.