| Challenge: | Recent approaches to quantization of Large Language Models (LLMs) have been widely adopted due to activation outliers, which degrade model performance especially at lower bit precision. |
| Approach: | They propose a new metric for quantization that strategically distributes outlier magnitudes across matrix dimensions via optimized diagonal operations. |
| Outcome: | The proposed framework achieves less than 1% accuracy drop in W4A4 quantization on the LLaMA-3-8B model and reduces the performance gap by 39.1% on the more challenging W2A4KV16 model. |
Similar Papers
LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for quantizing weights and activations of large language models suffer from non-negligible accuracy drops, especially on massive multitask language understanding. |
| Approach: | They propose a weight-activation quantization method that reconstructs the outputs of an intermediate Transformer block by leveraging low-rank weight-scaling matrices. |
| Outcome: | The proposed method reduces the complexity of the weight-activation quantization techniques while achieving high throughput and reducing inference costs. |
ATQ: Activation Transformation forWeight-Activation Quantization of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Quantization methods are available to solve the problem of high computational and storage costs for Large language models. |
| Approach: | They propose an INT8 weight-activation quantization method that can achieve lossless accuracy. |
| Outcome: | The proposed method can achieve lossless accuracy on OPT and LLaMA families. |
LRQuant: Learnable and Robust Post-Training Quantization for Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for post-training quantization (PTQ) are limited by the complexity of the quantization parameter and performance degradations when tested on unseen datasets. |
| Approach: | They propose a learnable smooth-based PTQ framework that allows for rapid adaptation during testing. |
| Outcome: | The proposed framework improves performance on unseen datasets and reduces memory constraints. |
ACBQ: Adaptive Cross-Block Quantization of Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for post-training quantization struggle to support weight–activation joint quantization and extreme low-bit weight quantization. |
| Approach: | They propose a framework that addresses weight–activation joint quantization and extreme weight quantization. |
| Outcome: | The proposed framework achieves superior performance under both W4A4 and highly aggressive W2 settings while incurring negligible additional computational overhead. |
VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on pushing weight-only quantization to extremely low-bit due to numerical representation limitations. |
| Approach: | They propose a vector-based quantization approach that pushes LLMs to extremely low-bit . they propose scalar-based weight quantization that reduces memory requirements and optimizes storage costs . |
| Outcome: | The proposed method reduces model quantization perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on mistral-7B, 4.41-7.34, on llaMA-3 on QA tasks on average. |
Achieving binary weight and activation for LLMs using Post-Training Quantization (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for quantizing large language models suffer from performance degradation when weights are quantized to 1 bit. |
| Approach: | They propose a post-training quantization framework with W(1+1)A(14) configuration . they propose utilizing Hessian-aware fine-grained grouping along with an EM-based quantization scheme . |
| Outcome: | The proposed method surpasses state-of-the-art (SOTA) LLM quantization baselines on W2A4 across multiple tasks. |
DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Quantization-aware Training (QAT) is a popular technique for reducing memory usage and improving computational efficiency in large language models. |
| Approach: | They propose a weight-decomposed low-rank quantization-aware training approach that integrates QAT with a group-specific quantization magnitude adjustment. |
| Outcome: | The proposed method outperforms the state-of-the-art method on LLaMA and LLama2 models. |
PE-QAT: Parameter-Efficient Quantization-Aware Training for Large Language Models (2026.acl-srw)
Copied to clipboard
| Challenge: | Quantization Aware Training (QAT) is expensive to train and unscalable to large models. |
| Approach: | They propose a parameter-efficient framework targeting per-channel 4-bit weight-activation quantization of large language models. |
| Outcome: | The proposed framework preserves accuracy within 0.11 percentage points of the full-precision baseline on Llama-2-7B zero-shot tasks while training only 1.26% of total parameters. |
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2024.findings-acl)
Copied to clipboard
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, Vikas Chandra
| Challenge: | Several post-training quantization methods have been shown to perform well down to 8-bits. |
| Approach: | They propose a data-free distillation method that leverages generations produced by the pre-trained model to quantize any generative model independent of its training data. |
| Outcome: | The proposed method outperforms SoTA PTQ and LLaMA models at low bit precision. |
Layer-Wise High-Impact Parameter Ratio Optimization in Post-Training Quantization for Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to quantize large language models suffer from significant accuracy loss at low bit-widths due to high-impact parameters. |
| Approach: | They propose a quadratic optimization framework that quantizes high-impact parameters to moderate bit-widths while quantizing low bit-wideths. |
| Outcome: | The proposed framework preserves high-impact parameters while preserving memory usage. |