Efficient LLM Comparative Assessment: A Product of Experts Framework for Pairwise Comparisons (2024.emnlp-main)
Copied to clipboard
| Challenge: | LLM-as-a-judge approaches are effective but cost scales quadratically with number of candidates, which has practical limitations. |
| Approach: | They propose a Product of Expert (PoE) framework for efficient LLM Comparative Assessment where individual comparisons are considered experts that provide information on a pair’s score difference. |
| Outcome: | The proposed framework can generate score predictions that correlate well with human judgements on multiple NLG tasks with as few as 2% of comparisons. |
Similar Papers
LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have enabled impressive zero-shot capabilities across various natural language tasks. |
| Approach: | They propose two ways to exploit the emergent abilities of large language models for NLG assessment. |
| Outcome: | The proposed methods improve performance and positional biases in comparisons between candidates. |
Finetuning LLMs for Comparative Assessment Tasks (2025.coling-main)
Copied to clipboard
| Challenge: | Automated assessment in natural language generation is a challenging task. |
| Approach: | They propose a framework for fine-tuning LLMs for comparative assessment to align the model’s output with the target distribution of comparative probabilities. |
| Outcome: | The proposed framework improves state-of-the-art performance while maintaining high performance with an efficient subset of comparisons. |
Agentic AI for Human Resources: LLM-Driven Candidate Assessment (2026.eacl-demo)
Copied to clipboard
Kamer Ali Yuksel, Abdul Basit Anees, Ashraf Hatim Elneima, Sanjika Hewavitharana, Mohamed Al-Badrashiny, Hassan Sawaf
| Challenge: | Current systems rely on keyword matching and shallow keyword-based screening, leading to missed opportunities and inconsistent evaluations. |
| Approach: | They propose a framework that uses Large Language Models to automate candidate assessment in recruitment. |
| Outcome: | The proposed framework outputs detailed assessment reports, candidate comparisons, and ranked recommendations that are transparent, auditable, and suitable for real-world hiring workflows. |
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2025.emnlp-main)
Copied to clipboard
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, Huan Liu
| Challenge: | Recent advances in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm . traditional methods of assessment and evaluation fail in dynamic and open-ended scenarios . |
| Approach: | They propose a paradigm where LLMs are leveraged to perform scoring, ranking, or selection for machine learning evaluation scenarios. |
| Outcome: | The proposed model-based judgment and evaluation paradigms are based on large language models and are compared to the current model-driven evaluation paradigm. |
Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat (2025.acl-long)
Copied to clipboard
| Challenge: | Evaluating large language models (LLMs) is a complex task. Pairwise ranking has emerged as state-of-the-art method to evaluate human preferences. |
| Approach: | They propose to use pairwise ranking to evaluate human preferences . they propose to evaluate the robustness of ranking algorithms in LLMs . |
| Outcome: | The proposed methods are based on the principles of effective ranking and the robustness of several ranking algorithms in the context of LLMs. |
From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation (2026.findings-acl)
Copied to clipboard
Pujun Zheng, Jiacheng Yao, Jinquan Zheng, Chenyang Gu, Guoxiu He, Jiawei Liu, Yong Huang, Tianrui Guo, Wei Lu
| Challenge: | Large language models (LLMs) are currently used to evaluate scientific papers by assigning an absolute score to each paper independently. |
| Approach: | They propose a comparison-native framework for paper evaluation that integrates comparison into both data construction and model learning. |
| Outcome: | The proposed framework achieves an average relative improvement of 21.8% over the strong baseline DeepReview-14B, while exhibiting robust generalization to five previously unseen datasets. |
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion (2023.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that open-source large language models (LLMs) exhibit diverse strengths and weaknesses due to variations in their architectures and training data. |
| Approach: | They propose a framework that leverages the diverse strengths of open-source large language models. |
| Outcome: | The proposed framework outperforms individual LLMs and baseline methods across various metrics, establishing a substantial performance gap. |
Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that evaluating NLG systems using pairwise comparisons is expensive as the number of human annotations grows linearly with k. |
| Approach: | They propose a framework to efficiently identify the top-ranked system by actively choosing system pairs for comparison using dueling bandit algorithms. |
| Outcome: | The proposed framework reduces human annotations by 80% on 13 NLG evaluation datasets spanning 5 tasks . |
LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient (2026.acl-long)
Copied to clipboard
Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Using generic and efficient benchmark generators, human annotators are limited by inefficiency . current benchmark generator methods rely on seed signals, leading to long cycles and high costs . |
| Approach: | They propose a framework to evaluate LLMs as generic benchmark generators and integrate them as BenchMaker. |
| Outcome: | The proposed framework achieves comparable performance to human-annotated benchmarks on most metrics. |
Praetor: A Fine-Grained Generative LLM Evaluator with Instance-Level Customizable Evaluation Criteria (2025.acl-long)
Copied to clipboard
Yongqi Leng, Renren Jin, Yue Chen, Zhuowen Han, Ling Shi, Jianxiang Peng, Lei Yang, Juesi Xiao, Deyi Xiong
| Challenge: | Existing evaluation methods are inadequate to evaluate large language models (LLMs). |
| Approach: | They propose a fine-grained generative LLM evaluator with instance-level customazable evaluation criteria that can be used to evaluate large language models. |
| Outcome: | The proposed model outperforms existing LLM evaluators and instruction-tuned LLMs on multiple benchmarks and sets new SOTA results. |