| Challenge: | Larger models or those trained on fewer tokens exhibit less quantization-induced degradation (QiD), while smaller, well-trained models face significant performance losses. |
| Approach: | They propose to use QiD to measure an LLM’s training levels and determine the number of training tokens required for fully training LLMs of various sizes. |
| Outcome: | The proposed scaling laws can predict the quantization performance of different-sized LLMs trained with tokens. |
Similar Papers
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2024.findings-acl)
Copied to clipboard
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, Vikas Chandra
| Challenge: | Several post-training quantization methods have been shown to perform well down to 8-bits. |
| Approach: | They propose a data-free distillation method that leverages generations produced by the pre-trained model to quantize any generative model independent of its training data. |
| Outcome: | The proposed method outperforms SoTA PTQ and LLaMA models at low bit precision. |
Why Do Some Inputs Break Low-Bit LLM Quantization? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Low-bit weight-only quantization reduces memory usage but disproportionately affects certain examples. |
| Approach: | They analyze quantization errors of 50 pairs of methods on large language models and test their hypothesis . |
| Outcome: | The proposed method reduces the memory footprint of large language models while maintaining reasonable performance across benchmarks. |
Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) require significant computational resources for deployment and use. |
| Approach: | They propose to use low-bit quantization methods to reduce memory footprint and increase inference rate to improve performance of Large Language Models. |
| Outcome: | The proposed methods can reduce the memory footprint and increase the inference rate of LLMs. |
A Comprehensive Evaluation of Quantization Strategies for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Quantization studies have focused on instruction-tuned LLMs, leaving their performance on other benchmarks unclear. |
| Approach: | They propose a framework to evaluate quantized large language models using four dimensions . they propose to reduce the bits needed for model weights or activations with minimal performance loss . |
| Outcome: | The proposed framework can retain comparable performance to non-quantized LLMs on most benchmarks. |
PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for sub 2-bit quantization introduce an extra 1-bit or more per weight. |
| Approach: | They propose a sub 2-bit post-training quantization method that enables weight quantization to 1.61-bit for the first time. |
| Outcome: | The proposed method reduces the upper bound of quantization error to 1.61-bit for the first time. |
LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for quantizing weights and activations of large language models suffer from non-negligible accuracy drops, especially on massive multitask language understanding. |
| Approach: | They propose a weight-activation quantization method that reconstructs the outputs of an intermediate Transformer block by leveraging low-rank weight-scaling matrices. |
| Outcome: | The proposed method reduces the complexity of the weight-activation quantization techniques while achieving high throughput and reducing inference costs. |
Achieving binary weight and activation for LLMs using Post-Training Quantization (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for quantizing large language models suffer from performance degradation when weights are quantized to 1 bit. |
| Approach: | They propose a post-training quantization framework with W(1+1)A(14) configuration . they propose utilizing Hessian-aware fine-grained grouping along with an EM-based quantization scheme . |
| Outcome: | The proposed method surpasses state-of-the-art (SOTA) LLM quantization baselines on W2A4 across multiple tasks. |
Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models? (2025.findings-acl)
Copied to clipboard
| Challenge: | Quantization-aware training of large language models reduces the precision of model parameters and reduces memory usage and energy consumption at inference time. |
| Approach: | They propose a method where models are first trained with 16-bit precision and then transition to 1.58-bit quantization-aware training. |
| Outcome: | The proposed training strategy reduces memory and energy consumption while maintaining model accuracy while reducing memory and inference time. |
Does quantization affect models’ performance on long-context tasks? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models support context windows exceeding 128K tokens, but this comes with significant memory requirements and high inference latency. |
| Approach: | They present the first systematic evaluation of quantized LLMs on tasks with long inputs and long-form outputs. |
| Outcome: | The proposed method preserves accuracy, while 4-bit methods lead to substantial losses . the results highlight the importance of a careful evaluation before deploying quantized LLMs . |
Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing quantisation methods mainly focus on 8-bit LLMs . a lack of scaling offsets in the quantisation process limits the use of LLM inference. |
| Approach: | They propose to use block quantisations to reduce scaling offsets in Large language models . they find that the block quantizations reduce scaling only from an arithmetic perspective . |
| Outcome: | The proposed methods reduce scaling offsets solely from an arithmetic perspective without additional treatments in the computational path. |