Toward Scalable Verifiable Reward: Proxy State-Based Evaluation for Multi-turn Tool-Calling LLM Agents (2026.acl-industry)
Copied to clipboard
Yun-Shiuan Chuang, Chaitanya Kulkarni, Alec M. Chiu, Avinash Thangali, Zijie Pan, Shivani Shekhar, Yirou Ge, Yixi Li, Uma Kona, Linsey Pang, Prakhar Mehrotra
| Challenge: | Existing agentic benchmarks rely on deterministic backends and are costly to build and iterate. |
| Approach: | They propose a framework that preserves final state-based evaluation without a deterministic database. |
| Outcome: | The proposed framework produces stable, model-differentiating rankings across families and inference-time reasoning efforts. |
Similar Papers
CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealistic settings but overlook reliability in real-world, user-facing applications. |
| Approach: | They propose a benchmark to evaluate consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. |
| Outcome: | The proposed benchmarks evaluate consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. |
Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges (2025.findings-acl)
Copied to clipboard
Hongru Wang, Wenyu Huang, Yufei Wang, Yuanhao Xi, Jianqiao Lu, Huan Zhang, Nan Hu, Zeming Liu, Jeff Z. Pan, Kam-Fai Wong
| Challenge: | Existing benchmarks that assess Language Models (LMs) as Language Agents (LAs) for tool use focus on stateless, single-turn interactions or partial evaluations, overlooking the inherent stateful nature of interactions in multi-turn applications. |
| Approach: | They propose a multi-turn dialogue dataset with stateful tool interactions considering the whole life cycle of tool use across six key tasks in three stages . they also build VirtualMobile – an embodied virtual mobile evaluation environment to simulate API calls and assess the robustness of the created APIs. |
| Outcome: | The proposed dataset evaluates 13 open- and closed-source LLMs and provides detailed analysis at each stage. |
Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents (2024.acl-long)
Copied to clipboard
Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Liujianfeng Liujianfeng, Ang Li, Jian Luan, Bin Wang, Rui Yan, Shuo Shang
| Challenge: | Existing benchmarks for LLM-based mobile agents are insufficient to evaluate their capabilities. |
| Approach: | They propose a benchmark to evaluate LLM-based mobile agents' planning capabilities . they expand UI operations by incorporating 103 APIs to accelerate task completion . |
| Outcome: | The proposed benchmarks are based on 103 collected APIs and real user queries . the data is categorized into three distinct groups: SAST, SAMT, and MAMT . |
MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains (2025.findings-naacl)
Copied to clipboard
Guoli Yin, Haoping Bai, Shuang Ma, Feng Nan, Yanchao Sun, Zhaoyang Xu, Shen Ma, Jiarui Lu, Xiang Kong, Aonan Zhang, Dian Ang Yap, Yizhe Zhang, Karsten Ahnert, Vik Kamath, Mathias Berglund, Dominic Walsh, Tobias Gindele, Juergen Wiest, Zhengfeng Lai, Xiaoming Simon Wang, Jiulong Shan, Meng Cao, Ruoming Pang, Zirui Wang
| Challenge: | Existing benchmarks focus on specific application scenarios, emphasizing task completion but failing to dissect the underlying skills that drive these outcomes. |
| Approach: | They propose a Massive Multitask Agent Understanding benchmark that evaluates LLMs across five domains and offline tasks. |
| Outcome: | The Massive Multitask Agent Understanding (MMAU) benchmark evaluates models across five domains including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. |
AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments (2025.acl-long)
Copied to clipboard
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang
| Challenge: | Large language models (LLMs) are promising foundations to build generally-capable agents . however, the community lacks a unified interactive framework that covers diverse environments for comprehensive evaluation of agents. |
| Approach: | They propose a framework that features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
| Outcome: | The proposed framework features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models have demonstrated remarkable performance across tasks. |
| Approach: | They propose a benchmark self-evolving framework to dynamically evaluate rapidly advancing Large Language Models. |
| Outcome: | The proposed framework extends existing benchmarks to extend models across tasks and tasks. |
MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing meta-evaluation benchmarks are static, outdated, and lacking in multilingual coverage. |
| Approach: | They propose a framework for curating more representative open-domain dialogue evaluation benchmarks . they leverage several LLMs to generate user-chatbot multilingual dialogues conditioned on varied seed contexts based on a state-of-the-art LLM . |
| Outcome: | The proposed framework exploits state-of-the-art LLMs to perform multilingual evaluations of open-domain chatbots. |
TelAgentBench: A Multi-faceted Benchmark for Evaluating LLM-based Agents in Telecommunications (2025.emnlp-industry)
Copied to clipboard
Sunwoo Lee, Daseong Jang, Dhammiko Arya, Gyoung-eun Han, Injee Song, SaeRom Kim, Sangjin Kim, Seojin Lee, Seokyoung Hong, Sereimony Sek, Seung-Mo Cho, Sohee Park, Sungbin Yoon, Wonbeom Jang, Eric Davis
| Challenge: | Large Language Models (LLMs) are becoming powerful agentic systems . generic benchmarks fail to assess realistic, non-English performance . |
| Approach: | They propose to evaluate five core agentic capabilities: Reasoning, Planning, Action (tool-use), Retrieval-Augmented Generation, and Instruction Following. |
| Outcome: | The evaluations reveal significant performance disparities between models that employ explicit reasoning and those that do not. |
ACEBench: A Comprehensive Evaluation of LLM Tool Usage (2025.findings-emnlp)
Copied to clipboard
Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Yuefeng Huang, Xiangcheng Liu, Wang Xinzhi, Wu Liu
| Challenge: | Existing benchmarks for evaluating LLMs’ tool usage face several limitations: limited evaluation scenarios, lacking assessments in real multi-turn dialogue contexts; narrow evaluation dimensions, with insufficient detailed assessments of how LLM use tools; and reliance on LLM or real API executions for evaluation, which introduces significant overhead. |
| Approach: | ACEBench is a benchmark for evaluating tool usage in Large Language Models . it categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. |
| Outcome: | ACEBench categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. |
AgentPro: Enhancing LLM Agents with Automated Process Supervision (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing frameworks lack explicit supervision during the reasoning process, which may lead to error propagation across reasoning chains. |
| Approach: | They propose a framework which automates process supervision for large language model agents by automatically generating step-level annotations and developing a process reward model based on these annotations. |
| Outcome: | The proposed framework outperforms existing agent-based methods on four datasets and achieves a 6.32% increase in accuracy. |