Papers with LongBench
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024.findings-acl)
Copied to clipboard
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Dongmei Zhang
| Challenge: | Existing approaches to compress prompts only leverage unidirectional context, causing suboptimal results. |
| Approach: | They propose a task-agnostic prompt compression method that takes tokens from context . they use a Transformer encoder to capture all essential information needed for prompt compression . |
| Outcome: | The proposed method is 3x-6x faster than existing prompt compression methods and faster than baselines. |
DisComp: A Two-Stage Prompt Optimization Framework Combining Task-Agnostic and Task-Aware Compression (2025.findings-naacl)
Copied to clipboard
| Challenge: | Extended prompts can lead to substantial computational overhead and increased hardware demands, limiting the scalability and efficiency of large language models. |
| Approach: | They propose a two-stage prompt compression framework that combines task-agnostic and task-based strategies to efficiently compress prompt length without compromising performance. |
| Outcome: | The proposed framework outperforms task-agnostic and task-specific compression methods on three benchmark datasets and is up to 6.56 faster at inference compared to the best token-level compression method. |
PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference (2026.findings-eacl)
Copied to clipboard
Krishna Teja Chitty-Venkata, Jie Ye, Siddhisanket Raskar, Anthony Kougkas, Xian Sun, Murali Emani, Venkatram Vishwanath, Bogdan Nicolae
| Challenge: | Large Language Models (LLMs) are exploding to large sizes, including GPT, LLaMA, and DeepSeek. |
| Approach: | They propose a fine-grained, structured KV cache pruning strategy that enhances the memory efficiency of vLLM’s PagedAttention. |
| Outcome: | The proposed method integrates seamlessly with PagedAttention without any modifications to its CUDA attention kernels. |
LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data (2025.findings-acl)
Copied to clipboard
| Challenge: | Long-context processing ability has emerged as a significant challenge for large language models. |
| Approach: | They propose a pipeline for synthesizing faithful long-context reasoning instruction datasets . they integrate ground truth and citation-based reasoning prompts integrating them . |
| Outcome: | The proposed pipeline eliminates distractions and improves reasoning chains. |
Mask Tokens as Prophet: Fine-Grained Cache Eviction for Efficient dLLM Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing cache eviction strategies for autoregressive language models fail to account for the role of mask tokens and specific characteristics in dLLMs. |
| Approach: | They propose a training-free cache eviction framework tailored to dLLMs that denies a fully masked sequence and allows parallel decoding at the expense of memory and computation. |
| Outcome: | The proposed framework reduces the cost of memory and cache eviction and improves efficiency by reducing allocation in intermediate layers and concentrating resources on prompt-preferring heads. |
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2024.acl-long)
Copied to clipboard
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li
| Challenge: | Large language models (LLMs) can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. |
| Approach: | They propose a bilingual, multi-task benchmark for long context understanding that extends context windows and more sophisticated memory mechanisms to improve models' long context capabilities. |
| Outcome: | The proposed model outperforms open-source models but struggles on longer contexts. |
Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing long-text evaluation benchmarks, such as L-Eval and LongBench, focus on QA and summarization tasks. |
| Approach: | They propose a length-adaptable benchmark for evaluating the long-context understanding of large language models. |
| Outcome: | The proposed benchmarks do not cover ultralong settings (100k+ tokens) and are difficult to evaluate across different length ranges. |
Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarios (2025.findings-naacl)
Copied to clipboard
| Challenge: | Long prompts contain redundant information and are sensitive to the position of key information in long context scenarios. |
| Approach: | They propose a training-free prompt compression framework that retains key information at token level while removing distracting tokens. |
| Outcome: | The proposed framework outperforms existing methods on long context benchmarks. |
CacheNotes: Task-Aware Key-Value Cache Compression for Reasoning-Intensive Knowledge Tasks (2026.eacl-long)
Copied to clipboard
| Challenge: | Current methods for integrating external knowledge into Large Language Models (LLMs) face limitations with broad, multi-source queries, while long-context models are computationally prohibitive. |
| Approach: | They propose a task-aware key-value cache compression method that generates a sequence of CPTs from a corpus and guides a one-time compression of the corpus into a compact, reusable KV cache. |
| Outcome: | The proposed method outperforms Retrieval-Augmented Generation (RAG) on Question-Answering tasks and reduces latency by over 4. |
Mitigate Position Bias in LLMs via Scaling a Single Hidden States Channel (2025.findings-acl)
Copied to clipboard
Yijiong Yu, Huiqiang Jiang, Xufang Luo, Qianhui Wu, Chin-Yew Lin, Dongsheng Li, Yuqing Yang, Yongfeng Huang, Lili Qiu
| Challenge: | Long-context language models exhibit position bias, also known as "lost in the middle" research shows that even long-contemporary LLMs fail to utilize all context information effectively . |
| Approach: | They propose a method to mitigate position bias by scaling positional hidden states . they propose to use a channel of hidden states to modify positional Hidden states a LCLM's positional bias . |
| Outcome: | The proposed method can improve performance by 15.2% in a "lost in the middle" benchmark. |
ReMamba: Equip Mamba with Effective Long-Sequence Modeling (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Mamba models demonstrate superior inference efficiency and competitive performance on short-context tasks, but their capacity to comprehend long contexts is limited compared to transformer-based models. |
| Approach: | They propose a model which incorporates selective compression and adaptation techniques within a two-stage re-forward process, incurring minimal additional inference costs overhead. |
| Outcome: | The proposed model improves on the LongBench and L-Eval benchmarks by 3.2 and 1.6 points and attains performance almost on par with same-size transformer models. |
ARK: Answer-Centric Retriever Tuning via KG-augmented Curriculum Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) is a powerful framework for knowledge-intensive tasks, but its effectiveness in long-context scenarios is often bottlenecked by the retriever’s inability to distinguish sparse yet crucial evidence. |
| Approach: | They propose a framework that fine-tunes the retriever for Answer Alignment by identifying high-quality positive chunks by evaluating their sufficiency to generate the correct answer. |
| Outcome: | The proposed framework improves 14.5% over the base model and maintains strong efficiency for long-context RAG. |
XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks. however, their extensive memory requirements present significant challenges for deployment in resource-constrained environments. |
| Approach: | They propose a training-free framework that achieves ultra-low equivalent bit-width KV cache quantization. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on TruthfulQA and LongBench. |
Evolving Sparsity: Leveraging Token Importance Dynamics for Efficient LLM Decoding with Sparse Attention (2026.acl-long)
Copied to clipboard
| Challenge: | Efficient long-context inference remains a major challenge for large language models (LLMs), as the cost of attention computation during auto-regressive decoding grows linearly with the context length. |
| Approach: | They propose to model token importance as a dynamic process that evolves over decoding steps and propagates through model layers. |
| Outcome: | The proposed method outperforms baseline sparse attention methods and achieves speedups of up to 5.36 for attention latency and 2.33 for end-to-end decoding. |
MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing LCU benchmarks for large language models often result in prohibitively high evaluation costs . existing benchmarks exhibit significant redundancy, which means inefficiency in evaluation . |
| Approach: | They propose a data compression method tailored for long-text data with sparse information characteristics. |
| Outcome: | The proposed method reduces evaluation costs to 4.5% of the long-text benchmark LongBench . the proposed method is based on a long-term LCU benchmark with sparse information characteristics . |
StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing compression approaches prioritize tokens based on local saliency metrics to decouple prefill computation from decoding memory. |
| Approach: | They propose a structure-aware KV cache compression framework that prioritizes tokens based on local saliency metrics to decouple prefill computation from decoding memory. |
| Outcome: | The proposed framework preserves long-range dependencies and retrieval robustness. |
LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for cache compression are heuristic and lack dynamic budget allocation . cnn's john mccartney and johnny mccain present a new approach for cache eviction and dynamic budgets . |
| Approach: | They propose a unified framework for cache compression that minimizes information loss in transformer residual streams. |
| Outcome: | The proposed method consistently maintains top performance across task types. |
100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability? (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for long-context capability are too synthetic and do not represent the real world usage of LLMs. |
| Approach: | They propose a length-controllable, real-life reflective benchmark that disentangles baseline knowledge from long-context capabilities. |
| Outcome: | Experiments show that the proposed benchmarks disentangle baseline knowledge from long-context capabilities. |
FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compression (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current compression strategies, including token eviction and learned projections, often lead to biased representations and may require costly model retraining. |
| Approach: | They propose a training-free KV cache compression framework that equalizes the contribution of all tokens to the compressed representation. |
| Outcome: | The proposed framework ensures unbiased information retention in the KV cache. |
DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods rely on information entropy as the metric to compress lexical units, but ignore attention-critical tokens and information . recent advent of In-Context Learning (ICL), Chain-of-Thought (CoT), and Retrieval Augmented Generation (RAG) technologies has significantly invigorated the landscape of applications based on Large Language Models (LLMs). |
| Approach: | They propose a dynamic attention-aware approach to task-agnostic prompt compression . they integrate entropy and attention information to achieve fine-grained prompt compression. |
| Outcome: | Experiments show that the proposed approach improves across tasks and LLMs. |
Adaptive Zooming via Relevance-Informed Positional Resource Allocation for Training-free LLM Context Extension (2026.findings-acl)
Copied to clipboard
Hongbo Zhao, Huibin Wang, Bin Tang, Xianming Hu, Yihong Huang, Yijun Shen, Nuoyi Chen, Ping Li, Kai Zhang
| Challenge: | Existing training-free methods for extrapolating beyond training context lengths are semantics-agnostic . Existing methods that focus on relative token distances can indiscriminately blur semantically relevant and irrelevant tokens . |
| Approach: | They propose an adaptive positional zooming method that uses semantic relevance to extrapolate beyond training context lengths. |
| Outcome: | Experiments show that RiPRA outperforms existing training-free extrapolation methods . relevant tokens get higher positional resolution, while irrelevant tokens are compressed . |
You Only Use Reactive Attention Slice When Retrieving From Long Context (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing retrieval techniques for language models are limited due to their reliance on lexical similarity and are computationally expensive to train. |
| Approach: | They propose a training-free and fine-tuning-free attention-based retrieval technique that uses a reaction score heuristic to quantify how an LM’s self-attention “reacts” to a user query. |
| Outcome: | The proposed approach improves QA task accuracy by 15% and inference throughput by 31% compared to embedding-based retrieval. |
Structure-aware Domain Knowledge Injection for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Structure-aware Continual Pre-Training (SCPT) and Structure-Aware Supervised Fine-Tuning (SSFT) are two-stage strategies for knowledge injection and alignment that reduces the training corpus needs to 5% while achieving 100% of traditional knowledge injection performance. |
| Approach: | They propose a method to efficiently transform foundation Large Language Models into domain specialists by using two-stage strategies: Structure-aware Continual Pre-Training and Structure-Aware Supervised Fine-Tuning. |
| Outcome: | The proposed method significantly reduces the training corpus needs to a mere 5% while achieving 100% of traditional knowledge injection performance. |
Squeezed Attention: Accelerating Long Context Length LLM Inference (2025.acl-long)
Copied to clipboard
Coleman Richard Charles Hooper, Sehoon Kim, Hiva Mohammadzadeh, Monishwaran Maheswaran, Sebastian Zhao, June Paik, Michael W. Mahoney, Kurt Keutzer, Amir Gholami
| Challenge: | Emerging Large Language Models require long input context to perform complex tasks. |
| Approach: | They propose an algorithm to reduce the complexity of attention with respect to the fixed context length. |
| Outcome: | The proposed method reduces the complexity of attention from linear to logarithmic with respect to the fixed context length. |
Towards Economical Inference: Enabling DeepSeek’s Multi-Head Latent Attention in Any Transformer-based LLMs (2025.acl-long)
Copied to clipboard
Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Shenlixing Shenlixing, Chenzhan Chenzhan, Xipeng Qiu, Qi Zhang, Tao Gui
| Challenge: | Multi-head Latent Attention (MLA) is an innovative architecture designed to ensure efficient and economical inference by significantly compressing the Key-Value (KV) cache into a latent vector. |
| Approach: | They propose a data-efficient fine-tuning method for transitioning from MHA to MLA using a latent vector cache. |
| Outcome: | The proposed architecture reduces the KV cache size of Llama2-7B by 92.19%, with only 1% drop in LongBench performance. |
Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries. |
| Approach: | They propose a lookahead q-cache framework that generates low-cost pseudo lookaheaded queries to better approximate the true decoding-stage queries. |
| Outcome: | The proposed framework outperforms existing methods on LongBench and Needle-in-a-Haystack benchmarks and can be flexibly combined to yield further improvements. |