Kelly Marchisio, Saurabh Dash, Hongyu Chen, Dennis Aumiller, Ahmet Üstün, Sara Hooker, Sebastian Ruder
| Challenge: | Quantization is widely used to improve inference speed and deployment of large language models. |
| Approach: | They conduct a thorough analysis of quantized multilingual LLMs . they find language disparately affected by quantization, non-Latin script languages worst . authors urge consideration of multilingual performance as evaluation criterion for efficient models . |
| Outcome: | The results show that quantization has harmful effects on human evaluation . language performance is disparately affected by quantization, the authors say . |
Similar Papers
A Comprehensive Evaluation of Quantization Strategies for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Quantization studies have focused on instruction-tuned LLMs, leaving their performance on other benchmarks unclear. |
| Approach: | They propose a framework to evaluate quantized large language models using four dimensions . they propose to reduce the bits needed for model weights or activations with minimal performance loss . |
| Outcome: | The proposed framework can retain comparable performance to non-quantized LLMs on most benchmarks. |
Calibrating Beyond English: Language Diversity for Better Quantized Multilingual LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing quantization methods typically use small, English-only calibration sets . however, their impact on multilingual models remains underexplored . |
| Approach: | They evaluate eight calibration settings across two quantizers on data from 10 different languages. |
| Outcome: | The results show that tailoring calibration sets to the evaluation language yields the largest improvements for individual languages, underscoring the importance of linguistic alignment. |
Does quantization affect models’ performance on long-context tasks? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models support context windows exceeding 128K tokens, but this comes with significant memory requirements and high inference latency. |
| Approach: | They present the first systematic evaluation of quantized LLMs on tasks with long inputs and long-form outputs. |
| Outcome: | The proposed method preserves accuracy, while 4-bit methods lead to substantial losses . the results highlight the importance of a careful evaluation before deploying quantized LLMs . |
When Quantization Affects Confidence of Large Language Models? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have shown that quantization compromises performance and exacerbates biases in Large Language Models. |
| Approach: | They propose an explanation for quantization loss based on confidence levels . they propose a range of efficient compression and acceleration methods including quan-tization . |
| Outcome: | The proposed methods show that quantization decreases confidence regarding true labels and that it exacerbates biases across different scales. |
Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) require significant computational resources for deployment and use. |
| Approach: | They propose to use low-bit quantization methods to reduce memory footprint and increase inference rate to improve performance of Large Language Models. |
| Outcome: | The proposed methods can reduce the memory footprint and increase the inference rate of LLMs. |
How Quantization Shapes Bias in Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | a systematic review of quantization's effects on model biases focuses on stereotypes, fairness, toxicity, and sentiment. |
| Approach: | They focus on weight and activation quantization strategies and examine their effects across bias types including stereotypes, fairness, toxicity, and sentiment. |
| Outcome: | The proposed method can reduce stereotypes and unfairness, but it tends to increase stereotypes in generative tasks. |
Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated multilingual capabilities, yet they are mostly English-centric due to the imbalanced training corpora. |
| Approach: | They extend the evaluation to real-world user queries and non-English-centric LLMs . they show that translation into English can boost LLM performance on NLP tasks . |
| Outcome: | The proposed evaluation extends to user queries and non-English-centric LLMs . it shows that translation into English can boost performance on NLP tasks, but not universally optimal . |
Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation? (2024.findings-eacl)
Copied to clipboard
Rishav Hada, Varun Gumma, Adrian Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, Sunayana Sitaram
| Challenge: | Large Language Models (LLMs) excel in various tasks, but their evaluation, especially in languages beyond the top 20, remains inadequate due to existing benchmarks and metrics limitations. |
| Approach: | They propose to use Large Language Models as evaluators to rank or score other models’ outputs by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages. |
| Outcome: | The proposed evaluation methods can be used to improve multilingual evaluation by calibrating them against 20K human judgments across three text-generation tasks, five metrics, and eight languages. |
How Reliable is Multilingual LLM-as-a-Judge? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLMs are a popular evaluation strategy, but their reliability in multilingual evaluation remains uncertain. |
| Approach: | They evaluate five models from different model families across five diverse tasks involving 25 languages. |
| Outcome: | The models perform poorly across languages and average Fleiss’ Kappa is 0.3 . |
Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs), yet translation errors in these benchmarks remain underexplored. |
| Approach: | They show how well machine-translated benchmarks match human span annotations on translations . they also show how strongly translation errors explain accuracy drops on translated benchmarks - a gap that is not addressed yet . |
| Outcome: | The proposed model matches human-level translations with human-language annotations on translations, but translation errors are associated with accuracy drops even after controlling for English correctness and source-side anomalies. |