Detecting Overflow in Compressed Token Representations for Retrieval-Augmented Generation (2026.eacl-srw)
Copied to clipboard
| Challenge: | Efficient long-context processing remains a challenge for large language models (LLMs) however, the limits of compressibility remain underexplored. |
| Approach: | They propose a method to characterize and detect token overflow in xRAG soft-compression by mapping long contexts into dense vectors that can be directly consumed by the model. |
| Outcome: | The proposed method identifies token overflow with query-agnostic saturation statistics but lacks the capability to detect it. |
Similar Papers
A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression (2025.acl-long)
Copied to clipboard
| Challenge: | gist-based context compression methods can achieve only slight performance loss on tasks like retrieval-augmented generation and long-document QA, but it faces challenges in tasks like synthetic recall. |
| Approach: | They propose two strategies to improve gist-based context compression in large language models. |
| Outcome: | The proposed methods can achieve only slight performance loss on retrieval-augmented generation and long-document QA tasks, but they face challenges in tasks like synthetic recall. |
CacheNotes: Task-Aware Key-Value Cache Compression for Reasoning-Intensive Knowledge Tasks (2026.eacl-long)
Copied to clipboard
| Challenge: | Current methods for integrating external knowledge into Large Language Models (LLMs) face limitations with broad, multi-source queries, while long-context models are computationally prohibitive. |
| Approach: | They propose a task-aware key-value cache compression method that generates a sequence of CPTs from a corpus and guides a one-time compression of the corpus into a compact, reusable KV cache. |
| Outcome: | The proposed method outperforms Retrieval-Augmented Generation (RAG) on Question-Answering tasks and reduces latency by over 4. |
Context Compression for Auto-regressive Transformers with Sentinel Tokens (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing Transformer-based LLMs have limited performance due to complexity of attention module . key-value cache is the major memory footprint and inference latency problem . |
| Approach: | They propose a plug-and-play approach that incrementally compresses token activation into compact ones . they also profile the benefit of context compression on improving the system throughout . |
| Outcome: | The proposed approach reduces memory footprint and inference latency by compressing tokens into compact ones. |
CompAct: Compressing Retrieved Documents Actively for Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to condense extensive documents with no loss of information are difficult to implement in real-world scenarios. |
| Approach: | They propose a framework that employs an active strategy to condense extensive documents without losing key information. |
| Outcome: | The proposed framework improves performance and compression rate on multi-hop question-answering benchmarks. |
DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokens (2025.findings-acl)
Copied to clipboard
Shaoshen Chen, Yangning Li, Zishan Xu, Yongqin Zeng, Shunlong Wu, Xinshuo Hu, Zifei Shan, Xin Su, Jiwei Tang, Yinghui Li, Hai-Tao Zheng
| Challenge: | Existing semantic vector-based compression methods do not account for the intrinsic information density variations between context chunks, instead allocating soft tokens uniformly across context chunk. |
| Approach: | They propose a method that leverages the LLM's intrinsic understanding of contextual relevance to guide compression. |
| Outcome: | The proposed method surpasses state-of-the-art methods on long context tasks. |
Refiner: Restructure Retrieved Content Efficiently to Advance Question-Answering Capabilities (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are limited by their parametric knowledge, leading to hallucinations in knowledge-extensive tasks. |
| Approach: | They propose an end-to-end extract-and-restructure paradigm that leverages a single decoder-only LLM to adaptively extract query-relevant contents verbatim along with the necessary context. |
| Outcome: | Experiments show that a trained Refiner outperforms state-of-the-art RAG and compressing approaches in multiple tasks. |
Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarios (2025.findings-naacl)
Copied to clipboard
| Challenge: | Long prompts contain redundant information and are sensitive to the position of key information in long context scenarios. |
| Approach: | They propose a training-free prompt compression framework that retains key information at token level while removing distracting tokens. |
| Outcome: | The proposed framework outperforms existing methods on long context benchmarks. |
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly lengthy and require longer prompts . this paper presents a coarse-to-fine prompt compression method to reduce cost and increase performance. |
| Approach: | They propose a coarse-to-fine prompt compression method that maintains semantic integrity under high compression ratios and a token-level iterative compression algorithm to better model the interdependence between compressed contents. |
| Outcome: | The proposed method yields state-of-the-art performance and allows for up to 20x compression with little performance loss over four datasets from different scenarios. |
TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | ChatGPT and GPT-4 are commercial large language models (LLMs) however, they may produce vague responses or incorrect answers in certain specialized domains. |
| Approach: | They propose a token compression scheme that uses summarization and semantic compression to reduce the token size of LLMs. |
| Outcome: | The proposed method reduces token size by doing summarization and semantic compression while reducing token size with only 1.6% accuracy drop. |
SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression (2026.acl-long)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) extends large language models with external knowledge, but it must balance limited effective context, redundant retrieved evidence, and the loss of fine-grained facts. |
| Approach: | They propose a hybrid RAG framework that uses natural-language snippets and semantic compression vectors to preserve passages in text form and compress remaining evidence into interpretable vectors for iterative evidence reranking. |
| Outcome: | The proposed framework improves answer relevance, answer correctness and semantic similarity across 9 datasets and 5 open-source LLMs. |