Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Existing methods for quantization-aware training and quantization for learning have limitations in dealing with accumulative quantization errors. |
| Approach: | They propose a method that minimizes accumulative quantization errors and outperforms baselines by distilling knowledge from a fine-tuned teacher network. |
| Outcome: | The proposed method minimizes accumulative quantization errors and outperforms baselines on the XGLUE benchmark. |
Similar Papers
On the Way to Lossless Compression of Language Transformers: Exploring Cross-Domain Properties of Quantization (2024.lrec-main)
Copied to clipboard
Nikita Martynov, Aleksei Goncharov, Gleb Kumichev, Evgeniy Egorov, Stanislav Vladimirovich Pavlov, Mikhail Sergeevich Durinov, Aleksandr Sergeevich Zuev, Egor Anatolievich Filimonov
| Challenge: | Modern Natural Language Processing models have a huge capacity, but this makes it difficult to employ. |
| Approach: | They propose a method to quantize at least 95% of Transformer weights without access to task-specific data so the drop in performance does not exceed 0.02%. |
| Outcome: | The proposed method quantizes 95% of Transformer weights and corresponding activations to INT8 without access to task-specific data so the drop in performance does not exceed 0.02%. |
Exploring Quantization for Efficient Pre-Training of Transformer Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Quantization has proven to be effective after pre-training and during fine-tuning, but its effects on pre-trainer performance have remained unexplored. |
| Approach: | They propose a linear quantization strategy to be applied during the pre-training of Transformers to improve model efficiency and stability. |
| Outcome: | The proposed method improves model efficiency, stability, and performance while maintaining language modeling ability. |
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2024.findings-acl)
Copied to clipboard
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, Vikas Chandra
| Challenge: | Several post-training quantization methods have been shown to perform well down to 8-bits. |
| Approach: | They propose a data-free distillation method that leverages generations produced by the pre-trained model to quantize any generative model independent of its training data. |
| Outcome: | The proposed method outperforms SoTA PTQ and LLaMA models at low bit precision. |
Dynamic Stashing Quantization for Efficient Transformer Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive performance on a range of Natural Language Processing (NLP) tasks. |
| Approach: | They propose a dynamic quantization strategy that reduces the amount of memory operations and reduces arithmetic cost by 20.95 on two translation tasks and three classification tasks. |
| Outcome: | The proposed model reduces the amount of arithmetic operations by 20.95 and the number of DRAM operations by 2.55 on two translation tasks and three classification tasks. |
LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for quantizing weights and activations of large language models suffer from non-negligible accuracy drops, especially on massive multitask language understanding. |
| Approach: | They propose a weight-activation quantization method that reconstructs the outputs of an intermediate Transformer block by leveraging low-rank weight-scaling matrices. |
| Outcome: | The proposed method reduces the complexity of the weight-activation quantization techniques while achieving high throughput and reducing inference costs. |
Compression of Generative Pre-trained Language Models via Quantization (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to compress generative pre-trained language models fail on generative tasks due to homogeneous word embeddings and limited memory. |
| Approach: | They propose a token-level contrastive distillation method to learn distinguishable word embeddings and a module-wise dynamic scaling method to make quantizers adaptive to different modules. |
| Outcome: | The proposed method outperforms the state-of-the-art compression methods on generative PLMs by a clear margin. |
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in transformer quantization have shown remarkable improvement in many Natural Language Processing tasks and beyond. |
| Approach: | They propose a novel quantization scheme for transformers that can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss. |
| Outcome: | The proposed methods achieve state-of-the-art results on the GLUE benchmark using BERT, while preserving memory and accuracy. |
When Quantization Affects Confidence of Large Language Models? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have shown that quantization compromises performance and exacerbates biases in Large Language Models. |
| Approach: | They propose an explanation for quantization loss based on confidence levels . they propose a range of efficient compression and acceleration methods including quan-tization . |
| Outcome: | The proposed methods show that quantization decreases confidence regarding true labels and that it exacerbates biases across different scales. |
A Frustratingly Easy Post-Training Quantization Scheme for LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Efficient inference is crucial for hyper-scale AI models, including large language models, as their parameter count continues to increase for enhanced performance. |
| Approach: | They propose a quantization scheme that fully utilizes the Transformer structure used in large language models to minimize the frequency of DRAM access while exploiting the parallelism of operations. |
| Outcome: | The proposed method minimizes the frequency of DRAM access while exploiting the parallelism of operations through a dense matrix format. |
Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling (2023.emnlp-main)
Copied to clipboard
| Challenge: | asymmetric outliers in transformer language models are a challenge for post-training quantization . we propose a framework for outlier suppression that can be seamlessly migrated into subsequent modules . |
| Approach: | They propose a framework for post-training quantization that includes the channel-wise shifting and scaling for concentration. |
| Outcome: | The proposed framework can be migrated into subsequent modules while maintaining equivalence. |