Papers with GPTQ
AFPQ: Asymmetric Floating Point Quantization for LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-bit weight quantization can save memory and accelerate inference. |
| Approach: | They propose asymmetric FP quantization which sets separate scales for positive and negative values. |
| Outcome: | The proposed method leads to large accuracy improvements and can be easily plugged into other quantization methods, including GPTQ and AWQ, for better performance. |
When Quantization Affects Confidence of Large Language Models? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have shown that quantization compromises performance and exacerbates biases in Large Language Models. |
| Approach: | They propose an explanation for quantization loss based on confidence levels . they propose a range of efficient compression and acceleration methods including quan-tization . |
| Outcome: | The proposed methods show that quantization decreases confidence regarding true labels and that it exacerbates biases across different scales. |
Calibrating Beyond English: Language Diversity for Better Quantized Multilingual LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing quantization methods typically use small, English-only calibration sets . however, their impact on multilingual models remains underexplored . |
| Approach: | They evaluate eight calibration settings across two quantizers on data from 10 different languages. |
| Outcome: | The results show that tailoring calibration sets to the evaluation language yields the largest improvements for individual languages, underscoring the importance of linguistic alignment. |
CoopQ: Cooperative Game Inspired Layerwise Mixed Precision Quantization for LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for large language models struggle when the average precision drops below four bits, limiting deployment on resourceconstrained devices such as mobiles, edge sensors, or standard GPUs. |
| Approach: | They propose a game-like game-inspired mixed-precision quantization method which translates these Shapley estimates into a binary quadratic optimization formulation, assigning either 2 or 4-bit precision to layers under strict memory constraints. |
| Outcome: | The proposed method reduces Perplexity by 20 – 80 % across average precisions spanning 4 bit down to 2 bit, compared to methods relying on isolated metrics. |
Revisiting Pruning vs Quantization for Small Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Compressing Small Language Models (SLMs) is particularly suited for resource-constrained devices, but their compression dynamics remain underexplored compared to Large Language Model (LLMs). |
| Approach: | They evaluated post-training pruning and quantization methods across six SLMs from 0.5 to 3.8B, seven languages, and seven downstream tasks. |
| Outcome: | The proposed methods outperform pruning and quantization on six SLMs from 0.5 to 3.8B, seven languages, and seven downstream tasks. |