MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models (2025.emnlp-demos)
Copied to clipboard
Zhiwei Liu, Jielin Qiu, Shiyu Wang, Jianguo Zhang, Zuxin Liu, Roshan Ram, Haolin Chen, Weiran Yao, Shelby Heinecke, Silvio Savarese, Huan Wang, Caiming Xiong
| Challenge: | Existing evaluation frameworks suffer from limitations such as static task benchmarks, limited scope, and inadequate integration with practical applications. |
| Approach: | They propose an open-source, Model Context Protocol-based evaluation framework specifically tailored for comprehensive and systematic assessment of LLM-powered agents. |
| Outcome: | The proposed framework uncovers nuanced performance patterns and identify domain-specific strengths and weaknesses, providing valuable insights beyond traditional binary success metrics. |
Similar Papers
Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistant (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Auto-SLURP is a benchmark dataset for evaluating multi-agent frameworks powered by large language models. |
| Approach: | Auto-SLURP is a benchmark dataset aimed at evaluating LLM-based multi-agent frameworks . authors propose it extends original SLURP dataset by relabeling data and integrating simulated servers and external services. |
| Outcome: | The proposed dataset extends the original SLURP dataset for natural language understanding tasks. |
AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments (2025.acl-long)
Copied to clipboard
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang
| Challenge: | Large language models (LLMs) are promising foundations to build generally-capable agents . however, the community lacks a unified interactive framework that covers diverse environments for comprehensive evaluation of agents. |
| Approach: | They propose a framework that features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
| Outcome: | The proposed framework features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP Tools (2026.acl-long)
Copied to clipboard
WenHao Wang, Peizhi Niu, Zhao Xu, Zhaoyu Chen, Jian Du, Yaxin Du, Xianghe Pang, Keduan Huang, Yanfeng Wang, Qiang Yan, Siheng Chen
| Challenge: | Existing research on Large Language Models (LLMs) relies on few servers and lacks training support. |
| Approach: | They propose a web-agent-driven pipeline for large-scale server discovery, data synthesis, and model training that collects and filters data from 1166 servers and 11536 tools. |
| Outcome: | Empirical evidence shows that MCP-Flow generates higher quality instruction-function call pairs and higher agentic task performance than previous work. |
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments (2026.acl-long)
Copied to clipboard
Zhiheng Xi, Dingwen Yang, Jiaqi Liu, Jixuan Huang, Honglin Guo, Baodai Huang, Tinggang Chen, Qi Zhang, Zhonghang Lu, Chenyu Liu, Jiajun Sun, Jiazheng Zhang, Dingwei Zhu, Xin Guo, Junzhe Wang, Zhihao Zhang, Yuming Yang, Junjie Ye, Minghe Gao, Dongrui Liu, Jiaming Ji, Guohao Li, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing benchmarks evaluate agents in simplified, idealized settings, relying on pre-packaged tool interfaces, overlooking critical steps, and assume inputs are clean and fully specified. |
| Approach: | They propose a framework that evaluates language agents in simplified, idealized settings . they show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 . |
| Outcome: | Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 . |
Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models have demonstrated remarkable performance across tasks. |
| Approach: | They propose a benchmark self-evolving framework to dynamically evaluate rapidly advancing Large Language Models. |
| Outcome: | The proposed framework extends existing benchmarks to extend models across tasks and tasks. |
HypoEval: Hypothesis-Guided Evaluation for Natural Language Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing frameworks for LLM-as-a-judge use zero-shot setting without consulting any human input, which leads to low alignment, or fine-tune LLMs on labeled data, which requires a non-trivial number of samples. |
| Approach: | They propose a hypothesis-guided evaluation framework that uses a small corpus of human evaluations to generate more detailed rubrics for human judgments and incorporates a checklist-like approach to combine LLM’s assigned scores on each decomposed dimension to acquire overall scores. |
| Outcome: | The proposed framework outperforms existing frameworks in both human rankings and human scores with 30 human evaluations and fine-tunes LLMs on labeled data with 3 times more human evaluation by 11.95%. |
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models (2025.naacl-industry)
Copied to clipboard
| Challenge: | Typical evaluations of Large Language Models (LLMs) report a single accuracy metric per dataset, often derived from an optimized setup. |
| Approach: | They propose a framework for non-adversarial evaluation of large language models that evaluates models by repeatedly testing them on the same benchmarks in various setups. |
| Outcome: | The proposed framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. |
AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing static safety evaluation methods are ill-equipped to address dynamic nature of AI risks and evolving regulations, creating a critical safety gap. |
| Approach: | They propose a new paradigm of agentic safety evaluation reframing evaluation as a continuous and self-evolving process rather than a one-time audit. |
| Outcome: | The proposed framework shows a consistent decline in model safety as the evaluation hardens. |
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2025.emnlp-main)
Copied to clipboard
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, Huan Liu
| Challenge: | Recent advances in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm . traditional methods of assessment and evaluation fail in dynamic and open-ended scenarios . |
| Approach: | They propose a paradigm where LLMs are leveraged to perform scoring, ranking, or selection for machine learning evaluation scenarios. |
| Outcome: | The proposed model-based judgment and evaluation paradigms are based on large language models and are compared to the current model-driven evaluation paradigm. |
Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation (2026.acl-long)
Copied to clipboard
Jiaju Chen, Yuxuan Lu, Xiaojie Wang, Huimin Zeng, Jing Huang, Jiri Gesi, Ying Xu, Bingsheng Yao, Dakuo Wang
| Challenge: | Existing "LLM-as-a-judge" evaluation frameworks are limited by persona descriptions and are not generalizable to other tasks. |
| Approach: | They propose a framework that can automatically construct multiple evaluator personas with distinct dimensions from relevant text documents and instantiate LLM agents with the persona. |
| Outcome: | The proposed framework can believably simulate human evaluators . it extracts stakeholders' diverse perspectives from the provided research papers and constructs personas for the agents . |