Arena-lite: Efficient and Reliable Large Language Model Evaluation via Tournament-Based Direct Comparisons (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current benchmarks typically compare system outputs against baselines, but this method yields lower reliability than direct comparison. |
| Approach: | They propose to integrate tournament structure on top of head-to-head comparison. |
| Outcome: | The proposed model achieves higher reliability with fewer comparisons even with smaller datasets or weaker judges. |
Similar Papers
Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models (2026.acl-long)
Copied to clipboard
Yanbin Yin, Kun Zhou, Zhen Wang, Xiangdong Zhang, Yifei Shao, Shibo Hao, Yi Gu, Jieyuan Liu, Somanshu Singla, Tianyang Liu, Eric P. Xing, Zhengzhong Liu, Haojian Jin, Zhiting Hu
| Challenge: | closed-ended question-based benchmarks struggle with saturation as newer models emerge . crowd-sourced leaderboards rely on costly and slow human judges . |
| Approach: | They propose a framework that leverages collective intelligence from all large language models to evaluate each other. |
| Outcome: | a new framework enables a democratic, pairwise evaluation of all large language models . it achieves 97% correlation with human judgements, while significantly reducing the cost. |
FlagEval-Arena: A Side-by-Side Comparative Evaluation Platform for Large Language Models and Text-Driven AIGC (2025.acl-demo)
Copied to clipboard
Jing-Shu Zheng, Richeng Xuan, Bowen Qin, Zheqi He, Tongshuai.ren Tongshuai.ren, Xuejing Li, Jin-Ge Yao, Xi Yang
| Challenge: | a new evaluation platform for large language models and text-driven AIGCs is available for free. |
| Approach: | They propose an evaluation platform for side-by-side comparisons of large language models and text-driven AIGC systems. |
| Outcome: | a new evaluation platform for large language models and text-driven AIGC systems is available for free . the platform is more focused on the Chinese language and more models developed by Chinese institutes . |
Model Consistency as a Cheap yet Predictive Proxy for LLM Elo Scores (2025.emnlp-main)
Copied to clipboard
| Challenge: | a rapid proliferation of large language models (LLMs) makes it difficult to assess which models are best suited for specific tasks. |
| Approach: | They find that the consistency of an LLM's Elo score is 91% correlated with its own human-produced Elo scores. |
| Outcome: | a new method to evaluate large language models is needed to scale with the increasing number of models released . current best way is to measure model's Elo score by comparing it to other models in contests . a simple proxy for Elo scores can be computed cheaply without human data or prior knowledge . |
StructEval: Deepen and Broaden Large Language Model Assessment via Structured Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Current evaluations for large language models use a single-item assessment paradigm . current evaluations struggle to discern whether a model possesses the required capabilities or merely memorizes/guesses the answers to specific questions. |
| Approach: | They propose a framework to evaluate large language models using atomic test objectives. |
| Outcome: | The proposed evaluation framework resists data contamination and reduces interference of potential biases, and sheds light on the design of future principled and trustworthy LLM evaluation protocols. |
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models (2025.naacl-industry)
Copied to clipboard
| Challenge: | Typical evaluations of Large Language Models (LLMs) report a single accuracy metric per dataset, often derived from an optimized setup. |
| Approach: | They propose a framework for non-adversarial evaluation of large language models that evaluates models by repeatedly testing them on the same benchmarks in various setups. |
| Outcome: | The proposed framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. |
Chatbot Arena Estimate: towards a generalized performance benchmark for LLM capabilities (2025.naacl-industry)
Copied to clipboard
Lucas Spangher, Tianle Li, William F. Arnold, Nick Masiewicki, Xerxes Dotiwalla, Rama Kumar Pasumarthi, Peter Grabowski, Eugene Ie, Daniel Gruhl
| Challenge: | Existing benchmark aggregation methods, such as Elo-based systems, can be resource-intensive, public facing, and time-consuming. |
| Approach: | They propose a framework for aggregating performance across diverse benchmarks that generates a “Goodness” and a ‘Fastness” score. |
| Outcome: | The proposed framework achieves higher Pearson correlation with Chatbot Arena Elo scores than MMLU’s correlation with chatbot Arena scores, validating its reliability for real-world LLM evaluation. |
Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat (2025.acl-long)
Copied to clipboard
| Challenge: | Evaluating large language models (LLMs) is a complex task. Pairwise ranking has emerged as state-of-the-art method to evaluate human preferences. |
| Approach: | They propose to use pairwise ranking to evaluate human preferences . they propose to evaluate the robustness of ranking algorithms in LLMs . |
| Outcome: | The proposed methods are based on the principles of effective ranking and the robustness of several ranking algorithms in the context of LLMs. |
CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation protocols for text generation suffer from rating inconsistencies . lexical overlap-based metrics align poorly with human judgments . |
| Approach: | They propose a checklist-based evaluation framework that improves rating reliability via decomposed binary questions. |
| Outcome: | The proposed framework improves rating reliability by decomposing binary questions . it improves agreement across evaluator models by 0.45 and reduces score variance . human evaluation remains the gold standard, but it #, Equal contribution. |
FreeEval: A Modular Framework for Trustworthy and Efficient Evaluation of Large Language Models (2024.emnlp-demo)
Copied to clipboard
Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Zhengran Zeng, Wei Ye, Jindong Wang, Yue Zhang, Shikun Zhang
| Challenge: | Large language models (LLMs) have revolutionized natural language processing with impressive performance across various tasks. |
| Approach: | They propose a framework for automated evaluations of large language models . they open-source their code at https://github.com/WisdomShell/FreeEval . |
| Outcome: | The framework is open-source and can be used to develop and validate new evaluation methods. |
League of LLMs: A Benchmark-Free Paradigm for Mutual Evaluation of Large Language Models (2026.acl-long)
Copied to clipboard
Qianhong Guo, Wei Xie, Xiaofang Cai, Enze Wang, Shuoyoucheng Ma, Xiaobing Sun, Tian Xia, Kai Chen, Xiaofeng Wang, Baosheng Wang
| Challenge: | Large language models (LLMs) have shown exceptional capabilities across a wide range of tasks, but reliable evaluation remains a challenge due to data contamination, opaque operation, and subjective preferences. |
| Approach: | They propose a benchmark-free evaluation paradigm that organizes multiple LLMs into a self-governed league for multi-round mutual evaluation. |
| Outcome: | Experiments on eight mainstream LLMs in mathematics and programming show that the proposed model can distinguish capabilities while maintaining high internal ranking stability. |