DEBATE: Devil’s Advocate-Based Assessment and Text Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for evaluating the quality of machine-generated texts have a relatively low correlation with human performance. |
| Approach: | They propose an NLG evaluation framework based on multi-agent scoring system augmented with a concept of Devil’s Advocate. |
| Outcome: | The proposed evaluation framework outperforms the previous state-of-the-art methods in two meta-evaluation benchmarks in NLG evaluation, SummEval and TopicalChat. |
Similar Papers
Unveiling the Achilles’ Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have highlighted various neural metrics that align well with human evaluations. |
| Approach: | They propose a black-box adversarial framework that generates strong disagreements between human and victim evaluators. |
| Outcome: | The proposed framework can significantly improve the performance of human and victim evaluators. |
Evaluating the Performance of Large Language Models via Debates (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are evolving and impacting various fields . current methods for evaluation are based on fixed, domain-specific questions or rely on human input, making them unscalable. |
| Approach: | They propose a benchmarking framework based on debates between LLMs, judged by another LLM. |
| Outcome: | The proposed framework achieves rankings that align closely with popular rankings based on human input eliminating the need for costly crowdsourcing. |
M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown their potential to deliver human-like judgments. |
| Approach: | They propose a systematic LLM-based multi-agent framework for advanced LLM as-a-judge MT evaluation that integrates dimension-specific results into a final evaluation judgment. |
| Outcome: | The proposed framework outperforms existing LLM-as-a-judge methods and competes with state-of-the-art automatic metrics even when powered by a suboptimal model like GPT-4o mini. |
Ranking Over Scoring: Towards Reliable and Robust Automated Evaluation of LLM-Generated Medical Explanatory Arguments (2025.coling-main)
Copied to clipboard
Iker De la Iglesia, Iakes Goenaga, Johanna Ramirez-Romero, Jose Maria Villa-Gonzalez, Josu Goikoetxea, Ander Barrena
| Challenge: | Evaluating LLM-generated text has become a key challenge in domain-specific contexts like the medical field. |
| Approach: | They propose a method to evaluate LLM-generated medical explanatory arguments using Proxy Tasks and rankings to align results with human evaluation criteria. |
| Outcome: | The proposed evaluation method is robust against adversarial attacks, including the assessment of non-argumentative text. |
Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation (2026.acl-long)
Copied to clipboard
Jiaju Chen, Yuxuan Lu, Xiaojie Wang, Huimin Zeng, Jing Huang, Jiri Gesi, Ying Xu, Bingsheng Yao, Dakuo Wang
| Challenge: | Existing "LLM-as-a-judge" evaluation frameworks are limited by persona descriptions and are not generalizable to other tasks. |
| Approach: | They propose a framework that can automatically construct multiple evaluator personas with distinct dimensions from relevant text documents and instantiate LLM agents with the persona. |
| Outcome: | The proposed framework can believably simulate human evaluators . it extracts stakeholders' diverse perspectives from the provided research papers and constructs personas for the agents . |
Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation (D19-1)
Copied to clipboard
| Challenge: | Existing evaluation methods for natural language generation are inadequate . distinguishing machine-generated text is challenging even for human evaluators . |
| Approach: | They compare human-based evaluators with automated evaluation procedures . they find human evaluers do not correlate well with discriminative evalators . |
| Outcome: | The proposed evaluation methods are compared with a dozen state-of-the-art generators for online product reviews. |
Debate, Deliberate, Decide (D3): A Cost-Aware Adversarial Framework for Reliable and Interpretable LLM Evaluation (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing evaluation tools for Large Language Models (LLMs) are inconsistency, bias, and lack of transparent decision criteria. |
| Approach: | They propose a cost-aware, adversarial multi-agent framework that orchestrates structured debate among role-specialized agents to produce reliable and interpretable evaluations. |
| Outcome: | The proposed framework orchestrates structured debate among role-specialized agents to produce reliable and interpretable evaluations. |
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2025.emnlp-main)
Copied to clipboard
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, Huan Liu
| Challenge: | Recent advances in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm . traditional methods of assessment and evaluation fail in dynamic and open-ended scenarios . |
| Approach: | They propose a paradigm where LLMs are leveraged to perform scoring, ranking, or selection for machine learning evaluation scenarios. |
| Outcome: | The proposed model-based judgment and evaluation paradigms are based on large language models and are compared to the current model-driven evaluation paradigm. |
LLM Agents at the Roundtable: A Multi-Perspective and Dialectical Reasoning Framework for Essay Scoring (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new framework for automated essay scoring is needed to achieve multi-perspective understanding and judgment. |
| Approach: | They propose a roundtable essay scoring framework that performs precise and human-aligned scoring under a zero-shot setting. |
| Outcome: | The proposed framework outperforms previous zero-shot AES approaches by enabling collaboration among agents with diverse evaluation perspectives. |
A LLM-based Ranking Method for the Evaluation of Automatic Counter-Narrative Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for evaluating CNs are expensive, time-consuming, and subjective, but lack a universal truth and the lack of a 'universal truth' . |
| Approach: | They propose a model ranking pipeline based on pairwise comparisons of generated CNs from different models organized in a tournament-style format to improve the evaluation process. |
| Outcome: | The proposed method achieves a high correlation with human preference, with a score of 0.88, and compares chat, instruct, and base models, exploring their strengths and limitations. |