RelayAttention for Efficient Large Language Model Serving with Long System Prompts (2024.acl-long)
Copied to clipboard
| Challenge: | a long system prompt causes throughput/latency bottlenecks as the cost of generating the next token increases w.r.t the sequence length. |
| Approach: | They propose an attention algorithm that reads hidden state from DRAM exactly once for a batch of input tokens. |
| Outcome: | The proposed algorithm reduces the need for redundant memory accesses in existing algorithms. |
Similar Papers
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2024.acl-long)
Copied to clipboard
| Challenge: | Longer prompts introduce irrelevant and redundant information, which can weaken LLMs' performance. |
| Approach: | They propose a prompt compression tool that improves LLMs' perception of key information in input prompts by up to 21.4% with around 4x fewer tokens in GPT-3.5-Turbo. |
| Outcome: | The proposed solution improves performance and reduces costs and latency by up to 21.4% with around 4x fewer tokens in the NaturalQuestions benchmark. |
ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition (2024.acl-long)
Copied to clipboard
| Challenge: | Experiments show that ChunkAttention can speed up the self-attention kernel by 3.2-4.8 compared to the start-of-the-art implementation. |
| Approach: | They propose a prefix-aware self-attention module that can detect matching prompt prefixes across multiple requests and share their key/value tensors in memory at runtime. |
| Outcome: | The proposed module can speed up the self-attention kernel by 3.2-4.8 compared to the start-of-the-art implementation, with the length of the system prompt ranging from 1024 to 4096. |
Leveraging Self-Attention for Input-Dependent Soft Prompting in LLMs (2025.acl-short)
Copied to clipboard
| Challenge: | Large language models require fine-tuning, which is computationally expensive and challenging. |
| Approach: | They propose a method that generates soft prompts based on input tokens and attends different tokens with varying importance. |
| Outcome: | The proposed method is simple and efficient, keeping the number of trainable parameters small. |
LLM in a flash: Efficient Large Language Model Inference with Limited Memory (2024.acl-long)
Copied to clipboard
Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S. Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, Mehrdad Farajtabar
| Challenge: | Large language models (LLMs) have high computational and memory requirements, especially for devices with limited memory. |
| Approach: | They propose a method that stores model parameters in flash memory but brings them on demand to DRAM . authors propose two techniques to optimize for reading data in larger, more contiguous chunks . |
| Outcome: | The proposed method reduces the volume of data transferred from flash and reads data in larger, more contiguous chunks. |
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly lengthy and require longer prompts . this paper presents a coarse-to-fine prompt compression method to reduce cost and increase performance. |
| Approach: | They propose a coarse-to-fine prompt compression method that maintains semantic integrity under high compression ratios and a token-level iterative compression algorithm to better model the interdependence between compressed contents. |
| Outcome: | The proposed method yields state-of-the-art performance and allows for up to 20x compression with little performance loss over four datasets from different scenarios. |
Attention Tracker: Detecting Prompt Injection Attacks in LLMs (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are vulnerable to prompt injection attacks, where malicious inputs manipulate the model into ignoring original instructions and executing designated actions. |
| Approach: | They propose a training-free method that tracks attention patterns on instruction to detect prompt injection attacks without additional inference. |
| Outcome: | The proposed method shows an improvement of up to 10.0% over existing methods and performs well even on small LLMs. |
Don’t Generate, Classify! Low-Latency Prompt Optimization with Structured Complementary Prompt (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated strong performance across diverse tasks, but their performance varies significantly across different prompts. |
| Approach: | They propose a framework that reframes prompt engineering as a classification problem. |
| Outcome: | The proposed framework improves answer quality by up to 26.5% in win rate compared to prior methods while reducing latency by upto 1,956 times. |
Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities in comprehending and analyzing lengthy sequential inputs. |
| Approach: | They propose to implement ad-hoc solutions that enhance LLMs’ performance on long input sequences by up to 50% while reducing API cost and latency by up . to address this limitation, they propose to use three datasets and two tasks to analyze news categorization and sentence analysis to evaluate their models. |
| Outcome: | The proposed solutions significantly improve LLMs’ performance on long input sequences by up to 50% while reducing API cost and latency by up . to 93% and 50%, respectively. |
Deputy: Accelerating Large Language Model Inference with Dynamic Low-Rank Substitution (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing dynamic schemes such as early-exit and layer-drop reduce FLOPs but break batch processing or introduce KV-cache inconsistency. |
| Approach: | They propose a dynamic low-rank substitution framework that employs a lightweight decision module at each layer to dynamically determine the execution branch for different tokens. |
| Outcome: | The proposed model reduces computation by approximately 40% compared to the original dense model while outperforming existing baseline methods. |
Evolving Sparsity: Leveraging Token Importance Dynamics for Efficient LLM Decoding with Sparse Attention (2026.acl-long)
Copied to clipboard
| Challenge: | Efficient long-context inference remains a major challenge for large language models (LLMs), as the cost of attention computation during auto-regressive decoding grows linearly with the context length. |
| Approach: | They propose to model token importance as a dynamic process that evolves over decoding steps and propagates through model layers. |
| Outcome: | The proposed method outperforms baseline sparse attention methods and achieves speedups of up to 5.36 for attention latency and 2.33 for end-to-end decoding. |