Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLM-as-Judge frameworks are increasingly popular for AI evaluation, yet research findings on the relationship between models’ generation and judgment abilities remain inconsistent. |
| Approach: | They propose a self-reference-guided evaluation strategy that leverages a model’s own answers as references to strengthen the correlation between generation and judgment abilities. |
| Outcome: | The proposed approach strengthens the correlation between model generation and judgment abilities and provides a reliable proxy for model selection in evaluation tasks. |
Similar Papers
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2025.emnlp-main)
Copied to clipboard
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, Huan Liu
| Challenge: | Recent advances in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm . traditional methods of assessment and evaluation fail in dynamic and open-ended scenarios . |
| Approach: | They propose a paradigm where LLMs are leveraged to perform scoring, ranking, or selection for machine learning evaluation scenarios. |
| Outcome: | The proposed model-based judgment and evaluation paradigms are based on large language models and are compared to the current model-driven evaluation paradigm. |
Humans or LLMs as the Judge? A Study on Judgement Bias (2024.emnlp-main)
Copied to clipboard
| Challenge: | Proprietary models such as GPT-4, Claude, Gemini-Pro and others are being democratized to improve evaluations of LLMs. |
| Approach: | They propose a framework that is free from referencing groundtruth annotations for investigating **Misinformation Oversight Bias**, **Gender Bia**,**Authority Bia* and **Beauty Bia's** on LLM and human judges. |
| Outcome: | The proposed framework investigates **Misinformation Oversight Bias**, **Gender Bia**,**Authority Bia* and **Beauty Bia' on LLM and human judges. |
JuStRank: Benchmarking LLM Judges for System Ranking (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work has focused on instance-based evaluation of LLM judges, where a judge is evaluated over a set of responses, or response pairs, while being agnostic to their source systems. |
| Approach: | They propose to validate the quality of the LLM judge itself by comparing system scores to a human-based ranking. |
| Outcome: | The proposed model fails to validate the quality of the judge itself, ignoring critical factors affecting system-level ranking, such as a judge’s positive or negative bias towards certain systems. |
Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings (2025.acl-long)
Copied to clipboard
| Challenge: | Contextual evaluation is challenging for state-of-the-art judge models . evaluation criteria are often conditional and dependent on practitioner priorities . |
| Approach: | They propose a judge benchmark that evaluates large language models as judges in contexts . they use human annotations and model-based perturbations to build the benchmark . |
| Outcome: | The proposed benchmark aims to evaluate large language models in contexts with 2,000 challenging response pairs. |
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge (2025.emnlp-main)
Copied to clipboard
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason E Weston, Sainbayar Sukhbaatar
| Challenge: | Existing methods for improving large language models have focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. |
| Approach: | They propose an iterative Meta-Rewarding step where the model judges its own judgements and uses that feedback to refine its judgment skills. |
| Outcome: | The proposed model improves Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2 and 20.6% to 29.1% on Arena-Hard. |
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2025.acl-short)
Copied to clipboard
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
| Challenge: | Existing evaluations of NLP models with LLMs are based on human judgments . however, there are concerns about their validity and reproducibility in proprietary models . |
| Approach: | They evaluate 11 current LLMs for their ability to replicate annotations. they show substantial variance across models and datasets. |
| Outcome: | The proposed model can replicate human annotations on 20 NLP datasets and show substantial variance across models and datasets. |
The Progress Illusion: Revisiting meta-evaluation standards of LLM evaluators (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLM judges have gained popularity as an inexpensive and performant substitute for human evaluation. |
| Approach: | They revisit meta-evaluations of LLM evaluators under a setting that more closely aligns with practice by examining evaluers’ ability to distinguish test system pairs that are closer in capability. |
| Outcome: | The proposed meta-evaluation setting is significantly different from the use of human evaluations. |
The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are widely used in NLP research but there is no standard or rigorous procedure to determine whether they can replace humans. |
| Approach: | They propose a statistical procedure that requires only a modest subset of annotated examples to justify using LLM annotations. |
| Outcome: | The proposed procedure compares LLMs with open-source LLM annotators and judges on ten language and vision-language tasks. |
Improve LLM-as-a-Judge Ability as a General Ability (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies focus on generative judges, but only on their judge ability. |
| Approach: | They propose a method that leverages the generative and reasoning capabilities of large language models to evaluate LLM responses across diverse scenarios, providing accurate preference signals. |
| Outcome: | The proposed model performs on RewardBench with only 2% to 40% of the data required by other training frameworks. |
Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation (2026.acl-long)
Copied to clipboard
Jiaju Chen, Yuxuan Lu, Xiaojie Wang, Huimin Zeng, Jing Huang, Jiri Gesi, Ying Xu, Bingsheng Yao, Dakuo Wang
| Challenge: | Existing "LLM-as-a-judge" evaluation frameworks are limited by persona descriptions and are not generalizable to other tasks. |
| Approach: | They propose a framework that can automatically construct multiple evaluator personas with distinct dimensions from relevant text documents and instantiate LLM agents with the persona. |
| Outcome: | The proposed framework can believably simulate human evaluators . it extracts stakeholders' diverse perspectives from the provided research papers and constructs personas for the agents . |