ETOM: A Five-Level Benchmark for Evaluating Tool Orchestration within the MCP Ecosystem (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing benchmarks assess tools in isolation, overlooking challenges such as functional overlap and cross-server orchestration, which can lead to overly optimistic evaluations. |
| Approach: | They propose a five-level benchmark for evaluating multi-hop, end-to-end tool orchestration by LLM agents within a hierarchical Model-Context Protocol (MCP) ecosystem. |
| Outcome: | The proposed framework evaluates end-to-end tool orchestration by agents in hierarchical Model-Context Protocol (MCP) environments. |
Similar Papers
Securing the Tool Layer: A Threat Taxonomy and Runtime Defense Framework for Model Context Protocol Deployments (2026.acl-industry)
Copied to clipboard
| Challenge: | Model Context Protocol (MCP) is the dominant standard for connecting large language models to external tools, databases, and services. |
| Approach: | They propose a runtime security framework that performs real-time validation of MCP tool calls and responses. |
| Outcome: | The proposed framework reduces attack success rates from 74% to under 9% for tool poisoning and from 47% to under6% for indirect prompt injection via tool responses. |
UniToolBench: A Benchmark for Tool-Augmented LLMs in Cross-Domain, Universal Task Automation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing benchmarks that focus on manually curated tool graphs lack scalability and diversity across domains. |
| Approach: | They propose a large-scale, cross-domain benchmark to evaluate LLMs' ability to reason over and utilize interconnected tools for automation. |
| Outcome: | The proposed benchmark incorporates automated tool graph construction by formulating link prediction as a probabilistic task, instead of relying on categorical LLM outputs. |
ACEBench: A Comprehensive Evaluation of LLM Tool Usage (2025.findings-emnlp)
Copied to clipboard
Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Yuefeng Huang, Xiangcheng Liu, Wang Xinzhi, Wu Liu
| Challenge: | Existing benchmarks for evaluating LLMs’ tool usage face several limitations: limited evaluation scenarios, lacking assessments in real multi-turn dialogue contexts; narrow evaluation dimensions, with insufficient detailed assessments of how LLM use tools; and reliance on LLM or real API executions for evaluation, which introduces significant overhead. |
| Approach: | ACEBench is a benchmark for evaluating tool usage in Large Language Models . it categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. |
| Outcome: | ACEBench categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. |
MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models (2025.emnlp-demos)
Copied to clipboard
Zhiwei Liu, Jielin Qiu, Shiyu Wang, Jianguo Zhang, Zuxin Liu, Roshan Ram, Haolin Chen, Weiran Yao, Shelby Heinecke, Silvio Savarese, Huan Wang, Caiming Xiong
| Challenge: | Existing evaluation frameworks suffer from limitations such as static task benchmarks, limited scope, and inadequate integration with practical applications. |
| Approach: | They propose an open-source, Model Context Protocol-based evaluation framework specifically tailored for comprehensive and systematic assessment of LLM-powered agents. |
| Outcome: | The proposed framework uncovers nuanced performance patterns and identify domain-specific strengths and weaknesses, providing valuable insights beyond traditional binary success metrics. |
An Evaluation Mechanism of LLM-based Agents on Manipulating APIs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have remarkable capabilities across a variety of tasks, such as language, mathematics, coding, and etc. |
| Approach: | They propose to decompose tool use capability into seven aspects and form a thorough evaluation schema for generic agents. |
| Outcome: | The proposed agent acts like a super-APP and can manipulate API-based tools. |
Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Enterprise systems are crucial for enhancing productivity and strategic growth, but data is fragmented across multiple sources and access controls are complex. |
| Approach: | They propose a benchmark that simulates enterprise settings with 500 diverse tasks . they show that even the most capable models achieve only 41.8% task completion . |
| Outcome: | The proposed benchmark shows that even the most capable models achieve only 41.8% task completion. |
AutoPenBench: A Vulnerability Testing Benchmark for Generative Agents (2025.emnlp-industry)
Copied to clipboard
Luca Gioacchini, Alexander Delsanto, Idilio Drago, Marco Mellia, Giuseppe Siracusano, Roberto Bifulco
| Challenge: | LLM agents are promising for vulnerability testing, but lack benchmarks to evaluate and compare them. |
| Approach: | They propose an open-source benchmark for the evaluation of vulnerability testing agents that includes 33 tasks ranging from introductory exercises to actual vulnerable systems. |
| Outcome: | The proposed benchmark includes 33 tasks ranging from introductory exercises to actual vulnerable systems. |
Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivity (2025.acl-long)
Copied to clipboard
| Challenge: | Personalized tool utilization is essential for aligning large language models (LLMs) with user preference in interaction scenarios with various tools. |
| Approach: | They propose a key-point-based LLM evaluation method that mitigates biases by manually annotating key points for each test case and providing them to LLM as the reference. |
| Outcome: | The proposed method mitigates biases in the LLM-as-a-judge system by manually annotating key points for each test case and providing them to LLM as the reference. |
TEA-Bench: A Systematic Benchmarking of Tool-enhanced Emotional Support Dialogue Agent (2026.acl-long)
Copied to clipboard
| Challenge: | Existing ESC systems focus on affective support in text-only settings, ignoring how external tools can enable factual grounding and reduce hallucination in multi-turn emotional support. |
| Approach: | They propose a benchmark for evaluating tool-augmented agents in ESC with realistic emotional scenarios and an MCP-style tool environment. |
| Outcome: | The proposed benchmarks show that tool augmentation improves emotional support quality and reduces hallucination, but weaker models benefit only marginally. |
MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP Tools (2026.acl-long)
Copied to clipboard
WenHao Wang, Peizhi Niu, Zhao Xu, Zhaoyu Chen, Jian Du, Yaxin Du, Xianghe Pang, Keduan Huang, Yanfeng Wang, Qiang Yan, Siheng Chen
| Challenge: | Existing research on Large Language Models (LLMs) relies on few servers and lacks training support. |
| Approach: | They propose a web-agent-driven pipeline for large-scale server discovery, data synthesis, and model training that collects and filters data from 1166 servers and 11536 tools. |
| Outcome: | Empirical evidence shows that MCP-Flow generates higher quality instruction-function call pairs and higher agentic task performance than previous work. |