Papers with QAT
BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation (2024.acl-long)
Copied to clipboard
| Challenge: | Weight quantization has emerged as a popular solution to reduce memory and computational demands. |
| Approach: | They propose a framework that synergizes Quantization-Aware Training (QAT) with Knowledge Distillation (KD) to boost the performance of LLMs at sub-4-bit. |
| Outcome: | The proposed framework outperforms existing QAT methods on language understanding and complex reasoning benchmarks on sub-4-bit models. |
DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Quantization-aware Training (QAT) is a popular technique for reducing memory usage and improving computational efficiency in large language models. |
| Approach: | They propose a weight-decomposed low-rank quantization-aware training approach that integrates QAT with a group-specific quantization magnitude adjustment. |
| Outcome: | The proposed method outperforms the state-of-the-art method on LLaMA and LLama2 models. |
PE-QAT: Parameter-Efficient Quantization-Aware Training for Large Language Models (2026.acl-srw)
Copied to clipboard
| Challenge: | Quantization Aware Training (QAT) is expensive to train and unscalable to large models. |
| Approach: | They propose a parameter-efficient framework targeting per-channel 4-bit weight-activation quantization of large language models. |
| Outcome: | The proposed framework preserves accuracy within 0.11 percentage points of the full-precision baseline on Llama-2-7B zero-shot tasks while training only 1.26% of total parameters. |
Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | Quantization-aware training (QAT) is a promising method to lower the implementation cost and energy consumption. |
| Approach: | They propose a method for fast converging QAT of pre-trained Transformers using a layer-wise signal propagation method with the intact signal from the teacher. |
| Outcome: | The proposed method achieves superior accuracy with significantly lower fine-tuning iterations on well-known Transformers of natural language processing as well as computer vision compared to the state-of-the-art methods. |
L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Quantization-aware PEFT methods have been developed to reduce memory and computational costs associated with large language models. |
| Approach: | They propose a method that integrates Quantization-Aware Training (QAT) with LoRA to reduce memory overhead and improve model accuracy. |
| Outcome: | The proposed method significantly reduces QAT’s memory overhead while preserving the advantage of QAT in producing fully quantized LLMs with high accuracy. |
Rotate, Clip, and Partition: Towards W2A4KV4 Quantization by Integrating Rotation and Learnable Non-uniform Quantizer (2025.findings-emnlp)
Copied to clipboard
| Challenge: | RCP can compress LLaMA-2-7B to W2A4KV4 with a loss of only 2.84 WikiText2 PPL and 5.29 times reduced memory footprint. |
| Approach: | They propose a Quantization-Aware Training approach that uses extreme compression of LLMs . they propose 'rotate, clip, and partition' to learn non-uniform intervals and a GPU kernel supporting GEMV . |
| Outcome: | The proposed method can compress LLaMA-2-7B to W2A4KV4 with a loss of only 2.84 WikiText2 PPL and 5.29 times reduced memory footprint. |
Understanding and Improving Knowledge Distillation for Quantization Aware Training of Large Transformer Encoders (2022.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) has been used for quantization-aware training to improve the ability of a lightweight model with the transferred knowledge from the teacher. |
| Approach: | They propose two methods to improve attention recovery of quantized large Transformers by combining attention-map and attention-output losses. |
| Outcome: | The proposed methods achieve state-of-the-art accuracy for quantized large Transformer encoder models with sub-2-bit weight quantization. |
EfficientQAT: Efficient Quantization-Aware Training for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Quantization-aware training (QAT) is a low-bit training solution that requires substantial training resources. |
| Approach: | They propose an algorithm that reduces memory consumption by low-bit representations with minimal accuracy loss. |
| Outcome: | EfficientQAT achieves 2-bit Llama-2-70B model on single GPU in 41 hours . compared to previous methods, it obtains model with less than 3 points accuracy degradation . |
GenPTQ: Green Post-Training Quantization for Large-Scale ASR Models with Mixed-Precision Bit Allocation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale models with high memory and computation demands face significant challenges for deployment. |
| Approach: | They propose a weight-only quantization method that optimizes for weights and QAT to restore accuracy. |
| Outcome: | The proposed method achieves 89.1% model size reduction with only 0.8% increase in WER and completes optimization in just 15 seconds. |
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional proficiency in language-related tasks, but their deployment poses significant memory and storage requirements. |
| Approach: | They propose a method that optimizes rounding values and weight clipping within 200 steps. |
| Outcome: | The proposed method achieves exceptional results across 2 to 4 bits while maintaining low tuning costs and avoiding additional inference overhead. |
Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs (2026.acl-long)
Copied to clipboard
Binxing Xu, Hao Gu, Lujun Li, Hao Wang, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Xintong Yang, Chao Li, Sirui Han, Yike Guo
| Challenge: | Existing approaches to training LLMs at ultra-low precisions suffer from convergence instability and substantial training costs. |
| Approach: | They propose a progressive QAT framework with outlier channel splitting to address these issues . they use nested structure of integer quantization grids to enable a "train once, deploy any precision" paradigm . |
| Outcome: | The proposed framework outperforms baselines on both Llama2/3 and W2A16, with an 11 speedup over BF16. |