XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference (2024.findings-emnlp)
Copied to clipboard
Joao Monteiro, Étienne Marcotte, Pierre-Andre Noel, Valentina Zantedeschi, David Vazquez, Nicolas Chapados, Christopher Pal, Perouz Taslakian
| Challenge: | XC-Llama uses pre-trained decoder-only models to condition generation on reference text without the prompt. |
| Approach: | They propose a model that uses cross-attention to condition generation on reference text without the prompt. |
| Outcome: | The proposed models outperform prompt-based inference methods and reduce space footprint relative to standard KV caching by two orders of magnitude. |
Similar Papers
S3Prompt: Instructing the Model with Self-calibration, Self-recall and Self-aggregation to Improve In-context Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models have limitations in practical applications, such as unsupervised generation and recall of in-context examples. |
| Approach: | They propose a self-calibration, self-recall and self-aggregation prompt pipeline to solve these problems. |
| Outcome: | The proposed pipeline improves the performance of large language models without annotating datasets and model parameter updates. |
IAM: Efficient Inference through Attention Mapping between Different-scale LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a challenge due to their internal reasoning processes. |
| Approach: | They propose an algorithm that can optimize attention matrices by performing attention mapping between small and large LLMs. |
| Outcome: | The proposed framework can reduce KV cache usage by 22.1% and accelerate prefill by 15% without sacrificing performance. |
Causal2Vec: Improving Decoder-only LLMs as Embedding Models through a Contextual Token (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods modify attention mechanism to be bidirectional, undermining LLMs’ ability to extract semantic information acquired during pre-training. |
| Approach: | They propose a general-purpose embedding model that pre-encodes input text into a single Contextual token and then prepends it to the LLM's input sequence. |
| Outcome: | The proposed model improves performance of decoder-only large language models without altering their architectures or introducing significant computational overhead. |
A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent studies have shown that sharing key-value (KV) cache across layers is effective in efficient inference of large language models. |
| Approach: | They propose a unified framework that covers several recent methods and their novel variants to investigate cross-layer KV sharing. |
| Outcome: | The proposed framework achieves higher throughput and better performance when reducing the size of the key-value cache by 2 while maintaining competitive performance. |
HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Existing static compression methods suffer from coarse-grained caching and high I/O overhead. |
| Approach: | They propose a training-free dynamic compression framework that uses a sparse attention mechanism to categorize attention heads based on stability and similarity. |
| Outcome: | The proposed framework achieves state-of-the-art performance on long-context benchmarks and accelerates decoding by up to 3 compared to the original model with a 224K context. |
EpMAN: Episodic Memory AttentioN for Generalizing to Longer Contexts (2025.acl-long)
Copied to clipboard
Subhajit Chaudhury, Payel Das, Sarathkrishna Swaminathan, Georgios Kollias, Elliot Nelson, Khushbu Pahwa, Tejaswini Pedapati, Igor Melnyk, Matthew Riemer
| Challenge: | Recent advances in Large Language Models (LLMs) have yielded impressive successes on many language tasks, but efficient processing of long contexts remains a significant challenge. |
| Approach: | They propose a method for processing long contexts in an episodic memory module while holistically attending to semantically-relevant context chunks. |
| Outcome: | The proposed method outperforms baseline decoders on multiple long-context recall and question-answering benchmarks on 16k to 256k tokens. |
Faster In-Context Learning for LLMs via N-Gram Trie Speculative Decoding (2025.emnlp-main)
Copied to clipboard
| Challenge: | In-Context Learning (ICL) is a key method in prompt engineering, but its long retrieved contexts and limited token throughput will slow reasoning speeds. |
| Approach: | They propose a method that leverages the overlap between context and model output to generate drafts from the context. |
| Outcome: | The proposed method achieves the highest mean speedup on Vicuna-7B, Llama2-7B-Chat, and Llma3-8B-Instruct tasks. |
Long-Context Language Modeling with Parallel Context Encoding (2024.acl-long)
Copied to clipboard
| Challenge: | Existing long-context models degenerate with retrieved contexts. |
| Approach: | They propose a framework that can be applied to existing decoder-only LLMs for context expansion. |
| Outcome: | The proposed framework can be applied to any existing decoder-only LLMs for context expansion. |
DeMPT: Decoding-enhanced Multi-phase Prompt Tuning for Making LLMs Be Better Context-aware Translators (2024.emnlp-main)
Copied to clipboard
| Challenge: | Concatenating large language models are adapted to context-aware neural machine translation in a concatenated way . a recent paradigm shift has been witnessed in discourse-related challenges such as zero pronoun translation . |
| Approach: | They propose an alternative adaptation approach to make large language models discriminately model and utilize inter- and intra-sentence contexts. |
| Outcome: | The proposed approach outperforms concatenation mode and improves performance in discourse modeling. |
C3D: Enhancing LLM Reasoning via Confidence-Guided Contrastive Decoding (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models are prone to distraction by contextual information during reasoning tasks. |
| Approach: | They propose a decoding method that uses predicted logits to estimate the model's confidence. |
| Outcome: | The proposed method reveals how the model dynamically activates and adjusts its consideration of each premise as reasoning progresses. |