SCVQ: Sparse-Compensated Vector Quantization for Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing vector quantization methods incur inference overhead due to massive codebook storage and intensive index lookups. |
| Approach: | They propose a framework for vector quantization that incorporates a salience-aware weighted K-means clustering scheme with symmetry constraints to reduce codebook size and indexing costs. |
| Outcome: | The proposed framework achieves a perplexity of 5.78 on WikiText-2 for LLaMA-2-7B at 2-bit quantization while delivering a 1.4 speedup over existing baselines. |
Similar Papers
VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on pushing weight-only quantization to extremely low-bit due to numerical representation limitations. |
| Approach: | They propose a vector-based quantization approach that pushes LLMs to extremely low-bit . they propose scalar-based weight quantization that reduces memory requirements and optimizes storage costs . |
| Outcome: | The proposed method reduces model quantization perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on mistral-7B, 4.41-7.34, on llaMA-3 on QA tasks on average. |
CARVQ: Corrective Adaptor with Group Residual Vector Quantization for LLM Embedding Compression (2025.findings-emnlp)
Copied to clipboard
Dayin Gou, Sanghyun Byun, Nilesh Malpeddi, Gabrielle De Micheli, Prathamesh Vaste, Jacob Song, Woo Seong Chung
| Challenge: | Large Language Models typically rely on a large number of parameters for token embedding, leading to substantial storage requirements and memory footprints. |
| Approach: | They propose a corrective Adaptor with group Residual Vector Quantization that can be used to compress the embedding layer without requiring specialized hardware. |
| Outcome: | The proposed corrective adaptor can achieve lower average bitwidth-per-parameter while maintaining reasonable perplexity and accuracy compared to scalar quantization. |
MLWQ: Efficient Small Language Model Deployment via Multi-Level Weight Quantization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for efficient deployment of small language models face inefficient bit-width allocation and insufficient fine-grained quantization adjustments. |
| Approach: | They propose a weight quantization technique that facilitates efficient deployment of SLMs . they propose to combine inter-layer loss and intra-layer salience to achieve better allocation . |
| Outcome: | Experimental results show that multi-level weight quantization achieves competitive performance compared to state-of-the-art methods. |
ACBQ: Adaptive Cross-Block Quantization of Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for post-training quantization struggle to support weight–activation joint quantization and extreme low-bit weight quantization. |
| Approach: | They propose a framework that addresses weight–activation joint quantization and extreme weight quantization. |
| Outcome: | The proposed framework achieves superior performance under both W4A4 and highly aggressive W2 settings while incurring negligible additional computational overhead. |
ATQ: Activation Transformation forWeight-Activation Quantization of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Quantization methods are available to solve the problem of high computational and storage costs for Large language models. |
| Approach: | They propose an INT8 weight-activation quantization method that can achieve lossless accuracy. |
| Outcome: | The proposed method can achieve lossless accuracy on OPT and LLaMA families. |
LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for quantizing weights and activations of large language models suffer from non-negligible accuracy drops, especially on massive multitask language understanding. |
| Approach: | They propose a weight-activation quantization method that reconstructs the outputs of an intermediate Transformer block by leveraging low-rank weight-scaling matrices. |
| Outcome: | The proposed method reduces the complexity of the weight-activation quantization techniques while achieving high throughput and reducing inference costs. |
VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing quantization methods for large language models suffer performance degradation at ultra-low bit-widths due to key cache outliers. |
| Approach: | They propose a vector quantization method that suppresses outliers in the key cache and reduces memory access overhead. |
| Outcome: | The proposed method outperforms baseline quantization methods across long-context understanding and mathematical reasoning tasks while minimizing memory access overhead. |
D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximation (2026.findings-acl)
Copied to clipboard
Junlin Li, Shuangyong Song, Guodong DU, Ngai Wong, Xuebo Liu, Yongxiang Li, Min Zhang, Jing Li, Xuelong Li
| Challenge: | Existing methods for fine-tuned large language models fail on fine-scale datasets . large data scale amplifies delta parameter magnitude, singular values, and entropy, causing compression errors. |
| Approach: | They propose a training- and data-free delta compression method that captures dominant delta structure and compensates residual low-rank approximation to recover fine-grained details from smaller residual error. |
| Outcome: | The proposed method outperforms existing methods on large-scale datasets on dense and MoE architectures. |
MEMORY-VQ: Compression for Tractable Internet-Scale Memory (2024.naacl-short)
Copied to clipboard
Yury Zemlyanskiy, Michiel de Jong, Luke Vilnis, Santiago Ontanon, William Cohen, Sumit Sanghai, Joshua Ainslie
| Challenge: | Memory-based methods like LUMEN pre-compute token representations for retrieved passages to speed up inference. |
| Approach: | They propose a method to reduce storage requirements of memory-augmented models . they use a vector quantization variational autoencoder to compress token representations . |
| Outcome: | The proposed method achieves 16x compression rate with comparable performance on KILT benchmark. |
AFPQ: Asymmetric Floating Point Quantization for LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-bit weight quantization can save memory and accelerate inference. |
| Approach: | They propose asymmetric FP quantization which sets separate scales for positive and negative values. |
| Outcome: | The proposed method leads to large accuracy improvements and can be easily plugged into other quantization methods, including GPTQ and AWQ, for better performance. |