Papers by Eldar Kurtic
“Give Me BF16 or Give Me Death”? Accuracy-Performance Trade-Offs in LLM Quantization (2025.acl-long)
Copied to clipboard
| Challenge: | despite popularity of large language model quantization, there are significant accuracy-performance trade-offs associated with quantization formats. |
| Approach: | They evaluate popular quantization formats across academic benchmarks and real-world tasks . they also examine the difference in text generated by quantized models versus their uncompressed counterparts . |
| Outcome: | The proposed format is lossless across all model scales and incurs low accuracy degradation when properly tuned. |
The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models (2022.emnlp-main)
Copied to clipboard
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, Dan Alistarh
| Challenge: | Pre-trained Transformer models provide robust language representations which can be specialized on various tasks. |
| Approach: | They propose an efficient pruning method based on approximate second-order information that allows pruning weight blocks to be used for pruning. |
| Outcome: | The proposed method is the first to be applied at the BERT scale and significantly pushes the boundaries of the current sparse models with respect to all metrics: model size, inference speed and task accuracy. |
Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | a new benchmark for evaluating the mathematical reasoning on large language models is being developed . popularity of reasoning benchmarks is leading to performance saturation and training set contamination. |
| Approach: | They introduce a benchmark for evaluating the mathematical reasoning on large language models . they find that models struggle with Mathador-LM, scoring lower than average 3rd graders . |
| Outcome: | The proposed benchmark improves performance on large language models . it also reduces test-set leakage into training data, a new study shows . |