Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats (2026.acl-industry)
Copied to clipboard
Pengxiang Zhao, Hui-Ling Zhen, Xing Li, Han Bao, Weizhe Lin, Zhiyuan Yang, Yu Zi Wei, Xin Wang, Mingxuan Yuan, Xianzhi Yu, Zhenhua Dong
| Challenge: | Low-bit floating-point formats like MXFP and NVFP4 offer new opportunities for precision and efficiency. |
| Approach: | They evaluate HiFloat (HiF8 and HiF4), a family of floating-point formats tailored for Ascend NPUs. |
| Outcome: | The proposed formats excel with high-variance data and are compatible with state-of-the-art quantization frameworks. |
Similar Papers
AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing research shows unsatisfactory empirical results for microscaling (MX) floating-point (FP) formats. |
| Approach: | They propose a 4-bit asymmetric FP format that handles activation outliers without calibration. |
| Outcome: | The proposed format outperforms MXFP4 by 3% on VQA and rotation-based methods by 1.6% on CSQA. |
Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formats (2026.acl-long)
Copied to clipboard
| Challenge: | Microscaling Floating-Point (MXFP) is a low-precision format for large language models (LLMs). |
| Approach: | They conduct systematic evaluations of PTQ under Microscaling Floating-Point (MXFP) . they find that MXFP8 consistently achieves near-lossless performance . |
| Outcome: | The proposed method achieves near-lossless performance while MXFP4 introduces substantial accuracy degradation and remains challenging. |
Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing quantisation methods mainly focus on 8-bit LLMs . a lack of scaling offsets in the quantisation process limits the use of LLM inference. |
| Approach: | They propose to use block quantisations to reduce scaling offsets in Large language models . they find that the block quantizations reduce scaling only from an arithmetic perspective . |
| Outcome: | The proposed methods reduce scaling offsets solely from an arithmetic perspective without additional treatments in the computational path. |
“Give Me BF16 or Give Me Death”? Accuracy-Performance Trade-Offs in LLM Quantization (2025.acl-long)
Copied to clipboard
| Challenge: | despite popularity of large language model quantization, there are significant accuracy-performance trade-offs associated with quantization formats. |
| Approach: | They evaluate popular quantization formats across academic benchmarks and real-world tasks . they also examine the difference in text generated by quantized models versus their uncompressed counterparts . |
| Outcome: | The proposed format is lossless across all model scales and incurs low accuracy degradation when properly tuned. |
LLM-FP4: 4-Bit Floating-Point Quantized Transformers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing quantization solutions are integer-based and struggle with bit widths below 8 bits. |
| Approach: | They propose a method for quantizing weights and activations in large language models down to 4-bit floating-point values in a post-training manner. |
| Outcome: | The proposed method outperforms existing methods on common sense zero-shot reasoning tasks by 12.7 points. |
AFPQ: Asymmetric Floating Point Quantization for LLMs (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-bit weight quantization can save memory and accelerate inference. |
| Approach: | They propose asymmetric FP quantization which sets separate scales for positive and negative values. |
| Outcome: | The proposed method leads to large accuracy improvements and can be easily plugged into other quantization methods, including GPTQ and AWQ, for better performance. |
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | NVFP4 supports fine-grained block isolation, 4-bit quantization errors and mixed-precision approaches . ARCQuant boosts NVFO4 performance via Augmented Residual Channels . |
| Approach: | They propose a framework that boosts NVFP4 performance via Augmented Residual Channels. |
| Outcome: | ARCQuant boosts NVFP4 performance via Augmented Residual Channels . the proposed framework achieves state-of-the-art accuracy comparable to full-precision baselines compared to FP16 . |
Fast Matrix Multiplications for Lookup Table-Quantized LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | generative (tokenby-token) inference is memory-bound and requires a large amount of memory to perform. |
| Approach: | They propose a lookup table engine for weight-quantized large language models that uses offline restructuring of the quantized weight matrix to minimize bit manipulations associated with unpacking. |
| Outcome: | The proposed kernel can be 2-4x faster than existing GEMM kernels while achieving performance gains of 1.5 to 2 times. |
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional proficiency in language-related tasks, but their deployment poses significant memory and storage requirements. |
| Approach: | They propose a method that optimizes rounding values and weight clipping within 200 steps. |
| Outcome: | The proposed method achieves exceptional results across 2 to 4 bits while maintaining low tuning costs and avoiding additional inference overhead. |
Achieving binary weight and activation for LLMs using Post-Training Quantization (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for quantizing large language models suffer from performance degradation when weights are quantized to 1 bit. |
| Approach: | They propose a post-training quantization framework with W(1+1)A(14) configuration . they propose utilizing Hessian-aware fine-grained grouping along with an EM-based quantization scheme . |
| Outcome: | The proposed method surpasses state-of-the-art (SOTA) LLM quantization baselines on W2A4 across multiple tasks. |