| Challenge: | Existing approaches to NLP are sparsifying attention patterns or approximating the attention computation with kernel methods. |
| Approach: | They propose a method for dynamic contextual compression for decoder-only LMs. |
| Outcome: | The proposed method reduces the cost of self-attention to a fraction of typical time and space. |
Similar Papers
Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent transformer language models achieve outstanding results on many downstream tasks, but their enormous size often makes them impractical on memory-constrained devices. |
| Approach: | They propose an offline compression approach that reduces the complexity of the model by enabling collaboration between modules. |
| Outcome: | The proposed approach outperforms commonly used factorization-based offline compression methods on various NLP tasks. |
Context Compression for Auto-regressive Transformers with Sentinel Tokens (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing Transformer-based LLMs have limited performance due to complexity of attention module . key-value cache is the major memory footprint and inference latency problem . |
| Approach: | They propose a plug-and-play approach that incrementally compresses token activation into compact ones . they also profile the benefit of context compression on improving the system throughout . |
| Outcome: | The proposed approach reduces memory footprint and inference latency by compressing tokens into compact ones. |
In-Context Former: Lightning-fast Compressing Context for Large Language Model (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to reduce inference costs of transformer-based large language models entail quadratic complexity . et al., 2017): transformer-derived large language model performance is a major challenge. |
| Approach: | They propose a method that compresses long contexts into short soft prompts . they use the self-attention mechanism of the large model to extract and condense information . |
| Outcome: | The proposed method reduces compression costs by 68 to 112 times while achieving 90% of baseline performance. |
What Context Features Can Transformer Language Models Use? (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. |
| Approach: | They propose to use lexical and structural information to ablate usable information in transformer language models. |
| Outcome: | The proposed model improves when conditioning on contexts of thousands of previous tokens. |
Adapting Language Models to Compress Contexts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Transformer-based language models have a finite context window and expensive computational cost of processing long text documents. |
| Approach: | They propose to adapt pre-trained LMs into AutoCompressors to compress text into summary vectors . authors propose to use summary vector to speed up inference over long contexts based on a finite context window . |
| Outcome: | The proposed model can compress long contexts into summary vectors, which are accessible as soft prompts. |
Pretraining Context Compressor for Large Language Models with Embedding-Based Memory (2025.acl-long)
Copied to clipboard
| Challenge: | Efficient processing of long contexts in large language models is essential for real-world applications such as retrieval-augmented generation and in-context learning. |
| Approach: | They propose a decoupled compressor-LLM framework that preserves contextual information within condensed embedding representations. |
| Outcome: | The proposed framework outperforms baseline models in three domains and across eight datasets while adapting to different downstream LLMs. |
Extending Context Window of Large Language Models via Semantic Compression (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models rely on a quadratic computation to generate long texts . current models impose limitations on the length of text inputs . |
| Approach: | They propose a semantic compression method that extends the context window of large language models . the method reduces the semantic redundancy of long inputs before passing them to the LLMs . |
| Outcome: | The proposed method extends the context window of large language models across tasks . it exhibits consistent fluency in text generation while reducing associated computational overhead. |
QUITO-X: A New Perspective on Context Compression from the Information Bottleneck Theory (2025.findings-emnlp)
Copied to clipboard
Yihang Wang, Xu Huang, Bowen Tian, Yueyang Su, Lei Yu, Huaming Liao, Yixing Fan, Jiafeng Guo, Xueqi Cheng
| Challenge: | Existing methods for compressing context by removing redundant tokens are inconsistent with the objective of retaining the most important tokens when conditioning on a given query. |
| Approach: | They propose a method that uses information bottleneck theory to compress context . they propose to remove redundant tokens using metrics such as self-information or perplexity . |
| Outcome: | The proposed method achieves a 25% increase in compression rate compared to the state-of-the-art . |
Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarios (2025.findings-naacl)
Copied to clipboard
| Challenge: | Long prompts contain redundant information and are sensitive to the position of key information in long context scenarios. |
| Approach: | They propose a training-free prompt compression framework that retains key information at token level while removing distracting tokens. |
| Outcome: | The proposed framework outperforms existing methods on long context benchmarks. |
When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit excellent performance in various tasks, but memory requirements present a challenge when deploying on memory-limited devices. |
| Approach: | They propose a framework to compress LLM after quantization further, achieving about 2.2x compression ratio. |
| Outcome: | The proposed model can achieve 40% reduction in memory size with negligible loss in accuracy and inference speed. |