XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks. however, their extensive memory requirements present significant challenges for deployment in resource-constrained environments. |
| Approach: | They propose a training-free framework that achieves ultra-low equivalent bit-width KV cache quantization. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on TruthfulQA and LongBench. |
Similar Papers
Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to compress KV cache compromise precision or require extra data for calibration, limiting their practicality in LLM deployment. |
| Approach: | They propose a low-bit quantization technique based on tensor decomposition to effectively compress KV cache. |
| Outcome: | The proposed method reduces memory footprint and performance by 75% . it is compared with existing methods that compromise precision or require extra data for calibration . |
pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for building efficient large language models with sub 2-bit weights are lacking in accuracy and scalability. |
| Approach: | They propose a method that decouples parameters by splitting linear layers into two specialized branches. |
| Outcome: | The proposed method achieves state-of-the-art performance in extremely low-bit quantization. |
AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models require substantial storage space to perform tasks such as text generation and video generation. |
| Approach: | They propose to compress large language models using integer replacements for floating-point numbers, in a process known as Quantization. |
| Outcome: | The proposed model allows for quantization of up to 75% decoder layers with 1 bit while maintaining performance levels comparable to those of the models with floating parameters. |
Accurate KV Cache Quantization with Outlier Tokens Tracing (2025.acl-long)
Copied to clipboard
Yi Su, Yuechi Zhou, Quantong Qiu, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, Min Zhang
| Challenge: | Large Language Models (LLMs) require substantial computational resources during deployment. |
| Approach: | They propose a method to identify outlier tokens and exclude them from quantization . they find that the method can deliver a 6.4 times reduction in memory usage and a 2.5 times increase in throughput . |
| Outcome: | The proposed method delivers a 6.4 times reduction in memory usage and a 2.5 times increase in throughput under 2-bit quantization. |
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2024.findings-acl)
Copied to clipboard
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, Vikas Chandra
| Challenge: | Several post-training quantization methods have been shown to perform well down to 8-bits. |
| Approach: | They propose a data-free distillation method that leverages generations produced by the pre-trained model to quantize any generative model independent of its training data. |
| Outcome: | The proposed method outperforms SoTA PTQ and LLaMA models at low bit precision. |
HqeKV: Towards Hybrid Quantization and Eviction for KV Cache in Long-Context LLM Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | autoregressive inference requires repeated computation across transformer layers. |
| Approach: | They propose a hybrid compression framework built on both quantization and eviction . they propose varying importance metric and flexible conversion policies to reduce memory overhead . |
| Outcome: | The proposed framework outperforms state-of-the-art methods under memory constraints. |
More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression (2025.findings-emnlp)
Copied to clipboard
Jiebin Zhang, Dawei Zhu, Yifan Song, Wenhao Wu, Chuqiao Kuang, Xiaoguang Li, Lifeng Shang, Qun Liu, Sujian Li
| Challenge: | storing more tokens in the KV cache at lower precision can enhance the long-context performance of large language models. |
| Approach: | They propose a token-precision trade-off strategy to optimize KV cache compression . they also propose storing more tokens in the KV at lower precision . |
| Outcome: | The proposed method achieves an optimal point within the Information Bottleneck compared to standalone KV pruning or KV quantization. |
TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering (2025.acl-industry)
Copied to clipboard
| Challenge: | key-value caches in large language models consume memory, posing a major challenge for scalable deployment. |
| Approach: | They propose a training-free recipe for KV cache compression with quantization precision that adapts to error sensitivity across layers and a mean centering to eliminate separate outlier handling. |
| Outcome: | The proposed technique reduces the KV cache memory footprint to 27% of the original 16-bit baseline while achieving comparable accuracy. |
VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing quantization methods for large language models suffer performance degradation at ultra-low bit-widths due to key cache outliers. |
| Approach: | They propose a vector quantization method that suppresses outliers in the key cache and reduces memory access overhead. |
| Outcome: | The proposed method outperforms baseline quantization methods across long-context understanding and mathematical reasoning tasks while minimizing memory access overhead. |
MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing low-bit quantization methods often exhibit severe performance degradation on complex reasoning tasks. |
| Approach: | They propose a plug-and-play method that uses a key channel's intrinsic quantization difficulty and relevance to the query to identify and preserve critical key channels that need higher precision. |
| Outcome: | Experiments on complex reasoning datasets show that the proposed method outperforms low-bit methods at a substantially reduced memory footprint. |