Becoming Experienced Judges: Selective Test-Time Learning for Evaluators (2026.eacl-short)
Copied to clipboard
| Challenge: | Large language models and visionlanguage models are increasingly used as automatic evaluators. |
| Approach: | They propose a framework that allows evaluators to improve *sequentially* at inference time without additional training or external signals. |
| Outcome: | The proposed framework outperforms strong baselines in two pairwise comparisons. |
Similar Papers
LRBench and Judge-R1: Principled Evaluation and Training of LLM-Based Judges for Long-Context Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for evaluating large language models (LLMs) under long contexts are underexplored. |
| Approach: | They propose a large-scale benchmark for evaluating large language models (LLMs) that combines reinforcement learning with multi-turn search to enable grounded and principle-aware evaluation. |
| Outcome: | The proposed model outperforms single-turn baselines across domains and principles. |
Enhancing LLM-as-a-Judge through Active-Sampling-based Prompt Optimization (2025.acl-industry)
Copied to clipboard
| Challenge: | Suboptimal prompts can introduce biases, inconsistencies, and unreliable evaluations. |
| Approach: | They propose an active-sampling-based framework for automatic prompt optimization . they use a small, diverse subset of samples to guide prompt refinement . |
| Outcome: | The proposed framework outperforms baselines on four popular LLMs and three real-world datasets. |
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2025.emnlp-main)
Copied to clipboard
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, Huan Liu
| Challenge: | Recent advances in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm . traditional methods of assessment and evaluation fail in dynamic and open-ended scenarios . |
| Approach: | They propose a paradigm where LLMs are leveraged to perform scoring, ranking, or selection for machine learning evaluation scenarios. |
| Outcome: | The proposed model-based judgment and evaluation paradigms are based on large language models and are compared to the current model-driven evaluation paradigm. |
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2025.acl-short)
Copied to clipboard
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
| Challenge: | Existing evaluations of NLP models with LLMs are based on human judgments . however, there are concerns about their validity and reproducibility in proprietary models . |
| Approach: | They evaluate 11 current LLMs for their ability to replicate annotations. they show substantial variance across models and datasets. |
| Outcome: | The proposed model can replicate human annotations on 20 NLP datasets and show substantial variance across models and datasets. |
All Prompts Are Created Equal? Evaluating Robustness of LLM Judges Against Non-Adversarial Prompt Variations (2026.findings-acl)
Copied to clipboard
| Challenge: | Our work establishes new standards for assessing LLM judge reliability beyond simple accuracy metrics. |
| Approach: | They propose a diagnostic framework grounded in attribute verifiability that enables principled decisions about evaluation automation. |
| Outcome: | The proposed framework establishes new standards for assessing LLM judge reliability beyond simple accuracy metrics. |
Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLM-as-Judge frameworks are increasingly popular for AI evaluation, yet research findings on the relationship between models’ generation and judgment abilities remain inconsistent. |
| Approach: | They propose a self-reference-guided evaluation strategy that leverages a model’s own answers as references to strengthen the correlation between generation and judgment abilities. |
| Outcome: | The proposed approach strengthens the correlation between model generation and judgment abilities and provides a reliable proxy for model selection in evaluation tasks. |
JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation (2026.findings-acl)
Copied to clipboard
Zhichao Shi, Xuhui Jiang, Chengjin Xu, Cangli Yao, Shengjie Ma, Yinghan Shen, Zixuan Li, Jian Guo, Yuanzhuo Wang
| Challenge: | Current evaluation methods for large language models rely on static benchmarks . limited knowledge coverage and fixed difficulties hinder the targeted optimizations resulting in superficial evaluations of LLMs - a problem that has been addressed by JudgeAgent . |
| Approach: | They propose a knowledge-driven and dynamic evaluation framework for large language models . judgeAgent leverages LLM agents equipped with context graphs to traverse knowledge structures . |
| Outcome: | The proposed framework can achieve comprehensive evaluations and facilitate effective model iterations. |
Praetor: A Fine-Grained Generative LLM Evaluator with Instance-Level Customizable Evaluation Criteria (2025.acl-long)
Copied to clipboard
Yongqi Leng, Renren Jin, Yue Chen, Zhuowen Han, Ling Shi, Jianxiang Peng, Lei Yang, Juesi Xiao, Deyi Xiong
| Challenge: | Existing evaluation methods are inadequate to evaluate large language models (LLMs). |
| Approach: | They propose a fine-grained generative LLM evaluator with instance-level customazable evaluation criteria that can be used to evaluate large language models. |
| Outcome: | The proposed model outperforms existing LLM evaluators and instruction-tuned LLMs on multiple benchmarks and sets new SOTA results. |
Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown impressive capabilities in many tasks, including natural language understanding and generation. |
| Approach: | They propose a framework for adaptation with self-evaluation to improve selective prediction performance of large language models. |
| Outcome: | The proposed framework outperforms state-of-the-art selective prediction methods on QA datasets and improves the AUACC from 91.23% to 92.63% and AUROC from 74.61% to 80.25%. |
Improve LLM-as-a-Judge Ability as a General Ability (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies focus on generative judges, but only on their judge ability. |
| Approach: | They propose a method that leverages the generative and reasoning capabilities of large language models to evaluate LLM responses across diverse scenarios, providing accurate preference signals. |
| Outcome: | The proposed model performs on RewardBench with only 2% to 40% of the data required by other training frameworks. |