| Challenge: | Recent work has focused on instance-based evaluation of LLM judges, where a judge is evaluated over a set of responses, or response pairs, while being agnostic to their source systems. |
| Approach: | They propose to validate the quality of the LLM judge itself by comparing system scores to a human-based ranking. |
| Outcome: | The proposed model fails to validate the quality of the judge itself, ignoring critical factors affecting system-level ranking, such as a judge’s positive or negative bias towards certain systems. |
Similar Papers
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2025.emnlp-main)
Copied to clipboard
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, Huan Liu
| Challenge: | Recent advances in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm . traditional methods of assessment and evaluation fail in dynamic and open-ended scenarios . |
| Approach: | They propose a paradigm where LLMs are leveraged to perform scoring, ranking, or selection for machine learning evaluation scenarios. |
| Outcome: | The proposed model-based judgment and evaluation paradigms are based on large language models and are compared to the current model-driven evaluation paradigm. |
Humans or LLMs as the Judge? A Study on Judgement Bias (2024.emnlp-main)
Copied to clipboard
| Challenge: | Proprietary models such as GPT-4, Claude, Gemini-Pro and others are being democratized to improve evaluations of LLMs. |
| Approach: | They propose a framework that is free from referencing groundtruth annotations for investigating **Misinformation Oversight Bias**, **Gender Bia**,**Authority Bia* and **Beauty Bia's** on LLM and human judges. |
| Outcome: | The proposed framework investigates **Misinformation Oversight Bias**, **Gender Bia**,**Authority Bia* and **Beauty Bia' on LLM and human judges. |
Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference (2025.findings-naacl)
Copied to clipboard
| Challenge: | Evaluating and ranking the capabilities of different LLMs is crucial for understanding their performance and alignment with human preferences. |
| Approach: | They propose a system-level evaluation framework that ranks LLMs based on their alignment with human preferences. |
| Outcome: | The proposed framework aims to rank LLMs based on their performance and alignment with human preferences. |
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2025.acl-short)
Copied to clipboard
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
| Challenge: | Existing evaluations of NLP models with LLMs are based on human judgments . however, there are concerns about their validity and reproducibility in proprietary models . |
| Approach: | They evaluate 11 current LLMs for their ability to replicate annotations. they show substantial variance across models and datasets. |
| Outcome: | The proposed model can replicate human annotations on 20 NLP datasets and show substantial variance across models and datasets. |
CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used to judge code, but their reliability remains poorly understood. |
| Approach: | They propose a benchmark to evaluate Large Language Models as code judges . they find that small reasoning models outperform larger non-reasoning models . |
| Outcome: | The proposed benchmark evaluates LLM-as-a-Judge models across three coding tasks. |
LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation metrics for natural language generation are expensive and time-consuming. |
| Approach: | They propose a framework that utilizes LLMs to generate synthetic evaluation datasets . they propose meta-correlation to measure alignment between metric rankings and human benchmarks based on synthetic data . |
| Outcome: | The proposed framework achieves meta-correlations exceeding 0.9 in multilingual QA and replaces human judgment with synthetic evaluation datasets. |
Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Using large language models (LLMs) for evaluating natural language generation has gained traction . lm judges have low intra-rater reliability in their assigned scores, making it difficult to measure how good their judgments actually are. |
| Approach: | They show that large language models align more closely with human preferences than n-grams . they quantify this variance and compare them to other NLG tasks and benchmarks based on the results . |
| Outcome: | The proposed models align more closely with human preferences than n-gram or embedding-based metrics. |
Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat (2025.acl-long)
Copied to clipboard
| Challenge: | Evaluating large language models (LLMs) is a complex task. Pairwise ranking has emerged as state-of-the-art method to evaluate human preferences. |
| Approach: | They propose to use pairwise ranking to evaluate human preferences . they propose to evaluate the robustness of ranking algorithms in LLMs . |
| Outcome: | The proposed methods are based on the principles of effective ranking and the robustness of several ranking algorithms in the context of LLMs. |
An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4 (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have fine-tuned judge models based on open-source LLMs to evaluate the quality of other LLM. |
| Approach: | They propose to use open-source LLMs to evaluate Large Language Models (LLMs) their empirical results show that the models underperform GPT-4 in several dimensions . |
| Outcome: | The proposed models outperform GPT-4 on several dimensions including generalizability, fairness and adaptability. |
Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation (2026.acl-long)
Copied to clipboard
Jiaju Chen, Yuxuan Lu, Xiaojie Wang, Huimin Zeng, Jing Huang, Jiri Gesi, Ying Xu, Bingsheng Yao, Dakuo Wang
| Challenge: | Existing "LLM-as-a-judge" evaluation frameworks are limited by persona descriptions and are not generalizable to other tasks. |
| Approach: | They propose a framework that can automatically construct multiple evaluator personas with distinct dimensions from relevant text documents and instantiate LLM agents with the persona. |
| Outcome: | The proposed framework can believably simulate human evaluators . it extracts stakeholders' diverse perspectives from the provided research papers and constructs personas for the agents . |