Papers with LLM-as-a-judge
The Data Frontier for Large Language Models: Selection, Synthesis, and Tools (2026.acl-tutorials)
Copied to clipboard
| Challenge: | acquiring and curating high-quality training data remains a significant bottleneck . acquiring such high-quality data is a key challenge for researchers and practitioners . |
| Approach: | This tutorial provides a comprehensive and practical guide to the state-of-the-art in data research directions for LLMs. |
| Outcome: | The tutorial covers methods for curating the most valuable information from vast, noisy datasets and the synthetic data revolution. |
Feedback-Aware Prompt Optimization Framework for Generating Job Postings (2026.eacl-industry)
Copied to clipboard
| Challenge: | Creating high-quality job postings is time-consuming and requires significant time from hiring managers and recruiters. |
| Approach: | They propose a feedback-aware prompt optimization framework that automates high-quality job posting generation through iterative human-in-the-loop refinement. |
| Outcome: | The proposed framework shows high compliance rates and strong satisfaction scores in both automated and human evaluations. |
Becoming Experienced Judges: Selective Test-Time Learning for Evaluators (2026.eacl-short)
Copied to clipboard
| Challenge: | Large language models and visionlanguage models are increasingly used as automatic evaluators. |
| Approach: | They propose a framework that allows evaluators to improve *sequentially* at inference time without additional training or external signals. |
| Outcome: | The proposed framework outperforms strong baselines in two pairwise comparisons. |
FiNERweb: Datasets and Artifacts for Scalable Multilingual Named Entity Recognition (2026.findings-eacl)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is the task of identifying tokens that belong to a predefined set of classes such as "person" or "location" |
| Approach: | They propose a dataset-creation pipeline that scales the teacher-student paradigm to 91 languages and 25 scripts. |
| Outcome: | The proposed model achieves comparable or improved performance in English, Thai, and Swahili despite being trained on 19x less data than strong baselines. |
IDEAlign: Comparing Ideas of Large Language Models to Domain Experts (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models are increasingly used to produce open-ended, interpretive annotations. |
| Approach: | They propose to use LLM annotations to evaluate content and assess expert similarity . they propose to benchmark different similarity methods against human ratings . |
| Outcome: | The proposed method performs best but falls short of expert alignment . it is useful as a triage filter rather than a substitute for human review. |
Large Language Models for Persian-English Idiom Translation (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models have shown superior capabilities in translating figurative language compared to neural machine translation systems. |
| Approach: | They evaluate LLMs, NMTs and their combinations using PersianIdioms datasets . they find that automatic evaluation methods like BLEU and BERTScore are effective . |
| Outcome: | The proposed model performs better in both directions than other models. |
Humans or LLMs as the Judge? A Study on Judgement Bias (2024.emnlp-main)
Copied to clipboard
| Challenge: | Proprietary models such as GPT-4, Claude, Gemini-Pro and others are being democratized to improve evaluations of LLMs. |
| Approach: | They propose a framework that is free from referencing groundtruth annotations for investigating **Misinformation Oversight Bias**, **Gender Bia**,**Authority Bia* and **Beauty Bia's** on LLM and human judges. |
| Outcome: | The proposed framework investigates **Misinformation Oversight Bias**, **Gender Bia**,**Authority Bia* and **Beauty Bia' on LLM and human judges. |
Praetor: A Fine-Grained Generative LLM Evaluator with Instance-Level Customizable Evaluation Criteria (2025.acl-long)
Copied to clipboard
Yongqi Leng, Renren Jin, Yue Chen, Zhuowen Han, Ling Shi, Jianxiang Peng, Lei Yang, Juesi Xiao, Deyi Xiong
| Challenge: | Existing evaluation methods are inadequate to evaluate large language models (LLMs). |
| Approach: | They propose a fine-grained generative LLM evaluator with instance-level customazable evaluation criteria that can be used to evaluate large language models. |
| Outcome: | The proposed model outperforms existing LLM evaluators and instruction-tuned LLMs on multiple benchmarks and sets new SOTA results. |
The Lessons of Developing Process Reward Models in Mathematical Reasoning (2025.findings-acl)
Copied to clipboard
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin
| Challenge: | a recent study shows that process reward models can make mistakes, leading to wrong conclusions. |
| Approach: | They propose a consensus filtering mechanism that integrates MC estimation with LLM-as-a-judge to improve model performance and data efficiency. |
| Outcome: | The proposed model outperforms existing open-source alternatives and provides practical guidelines for future research. |
Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are powerful automatic evaluators for natural language generation (NLG) tasks, but their uncertainty may limit their deployment in many applications. |
| Approach: | They propose a conformal prediction framework that provides a prediction interval with coverage guarantees and a midpoint-based score as a low-bias alternative to raw model score and weighted average. |
| Outcome: | The proposed framework provides a prediction interval with coverage guarantees and a midpoint-based score as a low-bias alternative to raw model score and weighted average. |
ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents (2026.findings-acl)
Copied to clipboard
| Challenge: | Reward-guided search methods have shown potential in enhancing tool-using agents . however, there is a lack of reliable evaluation benchmarks for PRMs in tool-use settings . |
| Approach: | They propose a large-scale benchmark specifically designed to evaluate PRMs for tool-using agents. |
| Outcome: | The proposed benchmark shows that tool reward models perform better in tool-using environments. |
Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing red-teaming methods generate adversarial attacks to identify vulnerabilities, but they face slow performance, limited categorical diversity, and high resource demands. |
| Approach: | They propose a method that generates multiple adversarial prompt mutations per iteration and ranks them using scoring functions. |
| Outcome: | The proposed method achieves a 95% attack success rate and reduces time to a 90% ASR by 15.2%. |
Evaluating LLMs’ Assessment of Mixed-Context Hallucination Through the Lens of Summarization (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models generate coherent text and follow instructions across diverse tasks, but a critical challenge in scaling LLM applications is hallucination, where the generated content lacks factual grounding or deviates from the intended discourse context. |
| Approach: | They use summarization as a representative task to evaluate LLMs' capability in detecting mixed-context hallucinations, specifically distinguishing between factual and non-factual hallucinos. |
| Outcome: | The proposed model distinguishes between factual and non-factual hallucinations, and their performance bottlenecks. |
AgentRM: Enhancing Agent Generalization with Reward Modeling (2025.acl-long)
Copied to clipboard
Yu Xia, Jingru Fan, Weize Chen, Siyu Yan, Xin Cong, Zhong Zhang, Yaxi Lu, Yankai Lin, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing LLM-based agents have strong performance on held-in tasks, but their generalizability to unseen tasks remains poor. |
| Approach: | They propose a reward-based generalizable reward model to guide the policy model for effective test-time search. |
| Outcome: | The proposed agentRM outperforms existing agents on held-in tasks by 8.8 points on average. |
Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing literature on leveraging persona in large language models is disorganized and lacks a systematic taxonomy . leveraging peopleas has resurfaced as an ideal lens for adapting LLMs for specific contexts . |
| Approach: | They propose to categorize current research on leveraging persona in large language models . they propose to use a comprehensive survey to categorize existing studies . |
| Outcome: | The proposed framework is a promising framework for tailoring large language models to specific contexts. |
MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generation (2025.acl-long)
Copied to clipboard
María Andrea Cruz Blandón, Jayasimha Talur, Bruno Charron, Dong Liu, Saab Mansour, Marcello Federico
| Challenge: | Existing benchmarks focus on English or use translated data, which fails to capture cultural nuances. |
| Approach: | They propose to use a multilingual end-to-end Meta-Evaluation RAG benchmark MEMERAG to assess accuracy and faithfulness of RAG systems. |
| Outcome: | The proposed benchmark can identify improvements offered by advanced prompting techniques and LLMs. |
TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles (2026.acl-long)
Copied to clipboard
Yirong Zeng, Yufei Liu, Xiao Ding, Yutai Hou, Yuxian Wang, Wu Ning, Haonan Song, Dandan Tu, Qixun Zhang, Yuxiang He, Bibo Cai, Ting Liu
| Challenge: | Instruction Following (IF) is a core capability of LLMs, requiring strict adherence to diverse constraints. |
| Approach: | They propose a framework that uses tiny language models to evaluate instruction following . they propose to use a set of specialized tiny language model to provide rewards for soft constraints. |
| Outcome: | The proposed framework outperforms baseline models by 12% and speeds up training time by 3. |
Improving LLM-as-a-Judge Inference with the Judgment Distribution (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Using language models to scalably approximate human preferences on text quality (LLM-as-a-judge) is a standard practice applicable to many tasks. |
| Approach: | They propose to use LLM judges to approximate human preferences on text quality by using distributional output over judgment tokens. |
| Outcome: | The proposed method outperforms taking the mode (i.e. greedy decoding) in all evaluation settings, and incorporating risk aversion improves performance. |
PASTEL : Polarity-Aware Sentiment Triplet Extraction with LLM-as-a-Judge (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for extracting triplets of aspect terms and opinions are inadequate due to complexity of aspect-opinion interactions and implicit nature of sentiment dependencies in natural language. |
| Approach: | They propose a pipeline that decomposes the ASTE task into structured subtasks . they employ fine-tuned LLMs to separately extract the aspect and opinion terms . |
| Outcome: | The proposed pipeline outperforms existing baselines in the ASTE subtask. |
Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) offer new opportunities to enhance the annotation process, particularly for detecting label errors in existing datasets. |
| Approach: | They propose to use an ensemble of large language models to flag mislabeled examples by using an LLM-as-a-judge approach to detect label errors in existing datasets. |
| Outcome: | The proposed method improves label accuracy and consistency in large language models. |
RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Prior studies show that large language models can draft fluent reviews but they miss specific issues, show shallow analysis, and produce generic phrasing. |
| Approach: | They propose a task that targets actionable review feedback generation and places existing peer review rebuttal at the center of learning. |
| Outcome: | The proposed model improves on a large dataset that maps review segments to rebuttal segments that address them, with perspective labels and impact categories that order author uptake. |
LLM Multi-Agent Systems for Long Triple Set Data-to-Text Generation (2026.findings-acl)
Copied to clipboard
Chinonso Cynthia Osuji, Simon Mille, Mark Andrade, Jane Adkins, Ornait O’Connell, Elaine Uí Dhonnchadha, Bláithín Heffernan, Fírinne Nic an tSaoir, Anya Belz, Thiago Castro Ferreira, Brian Davis
| Challenge: | Existing data-to-text benchmarks that do not involve content selection feature short input-output pairs designed for sentence or paragraph-level generation with reference texts spanning only a few dozen tokens. |
| Approach: | They propose a system that generates multi-paragraph outputs in English and Irish . they compare a multi-agent configuration against a single-task variant . |
| Outcome: | The proposed framework generates multi-paragraph outputs in English and Irish . human evaluation and LLM-as-a-judge score better in both languages . |
MARCH: Multi-Agent Reinforced Check for Hallucination (2026.acl-long)
Copied to clipboard
Zhuo Li, Yupeng Zhang, Pengyu Cheng, Jiajun Song, Mengyu Zhou, Hao Li, Shujie Hu, Yu Qin, null Erchao.zec, Xiaoxi Jiang, null Guanjunjiang
| Challenge: | Existing methods to detect hallucinations suffer from inherent confirmation bias, where the verifier inadvertently reproduces the errors of the original generation. |
| Approach: | They propose a framework that enforces rigorous factual alignment by leveraging deliberate *information asymmetry* by combining a pipeline of three specialized agents: a Solver, a Proposer, and a Checker. |
| Outcome: | Extensive experiments across hallucination benchmarks demonstrate that MARCH substantially reduces hallucinism rates. |
GeoRC: A Benchmark for Geolocation Reasoning Chains (2026.acl-long)
Copied to clipboard
Mohit Talreja, Joshua Diao, Jim James, Radu Casapu, Tejas Santanam, Ethan Mendes, Alan Ritter, Wei Xu, James Hays
| Challenge: | Vision Language Models (VLMs) are good at recognizing the global location of a photograph but are startlingly bad at explaining which image evidence led to their location prediction. |
| Approach: | They propose a benchmark for geolocation reasoning chains based on the global location prediction task in the popular GeoGuessr game. |
| Outcome: | The proposed benchmark compares LLM-as-a-judge and VLM-As-jumble strategies against human scoring. |
Alignment Data Map for Efficient Preference Data Selection and Diagnosis (2026.findings-acl)
Copied to clipboard
| Challenge: | constructing high-quality preference datasets faces scalability challenges due to prohibitive cost and complexity of human annotation. |
| Approach: | They propose a tool to identify and select effective preference data by LLM-as-a-judge, explicit reward model, and reference-based approaches. |
| Outcome: | The proposed tool reduces annotation costs while preserving alignment effectiveness. |
Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks suffer from semantic drift and context loss, which can lead to misleading performance metrics. |
| Approach: | They propose a fully automated framework to enable translation of large language models . they propose to use universal self-improvement and multi-round ranking methods to improve translation quality . |
| Outcome: | The proposed framework surpasses existing benchmarks in eight languages and improves translation quality across multilingual domains. |