MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents (2026.acl-long)
Copied to clipboard
Pengxiang Zhao, Guangyi Liu, Yaozhen Liang, Weiqing He, Zhengxi Lu, WenHao Wang, Yuehao Huang, Yuxiang Chai, Zhaolu Kang, Yaxuan Guo, Hao Wang, Kexin Zhang, Liang Liu, Yong Liu
| Challenge: | Shortcuts such as APIs and deep-links have emerged as efficient complements to flexible GUI operations, but systematic evaluation of GUI–shortcut hybrid agents remains underexplored. |
| Approach: | They propose a benchmark that evaluates GUI-shortcut hybrid agents with a specific focus on the mobile domain. |
| Outcome: | MAS-Bench evaluates agent's ability to generate shortcuts by discovering and creating reusable, low-cost workflows. |
Similar Papers
Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents (2024.acl-long)
Copied to clipboard
Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Liujianfeng Liujianfeng, Ang Li, Jian Luan, Bin Wang, Rui Yan, Shuo Shang
| Challenge: | Existing benchmarks for LLM-based mobile agents are insufficient to evaluate their capabilities. |
| Approach: | They propose a benchmark to evaluate LLM-based mobile agents' planning capabilities . they expand UI operations by incorporating 103 APIs to accelerate task completion . |
| Outcome: | The proposed benchmarks are based on 103 collected APIs and real user queries . the data is categorized into three distinct groups: SAST, SAMT, and MAMT . |
MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments (2026.acl-long)
Copied to clipboard
Quyu Kong, Xu Zhang, Zhenyu Yang, Nolan Gao, Chen Liu, Panrong Tong, Chenglin Cai, Hanzhang Zhou, Jianan Zhang, Liangyu Chen, Zhidan Liu, Steven Hoi, Yue Wang
| Challenge: | AndroidWorld is the dominant mobile GUI agent evaluation benchmark, but its success rates are low . despite reproducible emulator environment, it lacks key application categories such as e-commerce and enterprise communication. |
| Approach: | They propose a benchmark for mobile GUI agents that reflects real-world usage through long-horizon, cross-application workflows. |
| Outcome: | The proposed framework achieves over 90% success rates, while AndroidWorld is the dominant benchmark. |
LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces (2026.findings-acl)
Copied to clipboard
Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Shenglin Zhang, Kaipeng Zhang
| Challenge: | Existing benchmarks for agentic programming in long-horizon command-line interface tasks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics. |
| Approach: | They propose a benchmark to evaluate agentic capabilities across long-horizon command-line interface tasks. |
| Outcome: | The proposed benchmarks cover four engineering categories: from scratch, feature addition, bug fixing, and refactoring. |
TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments (2025.findings-acl)
Copied to clipboard
Yuheng Lu, Qian Yu, Hongru Wang, Zeming Liu, Wei Su, Yanping Liu, Yuhang Guo, Maocheng Liang, Yunhong Wang, Haifeng Wang
| Challenge: | Existing GUI agents struggle to adapt to dynamic and interconnected nature of real-world digital environments, authors show . |
| Approach: | They propose a benchmark to evaluate the transferability of GUI agents across three key dimensions . transBench includes 15 app categories with diverse functionalities . |
| Outcome: | The proposed benchmark shows that existing GUI agents struggle to adapt to dynamic, interconnected environments. |
MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in mobile Graphical User Interface (GUI) agents highlight the growing need for comprehensive evaluation benchmarks. |
| Approach: | They propose an online benchmark with 1080 tasks from 80 Chinese apps that measures task execution, complex reasoning, noise robustness and auto-eval framework with a reset mechanism. |
| Outcome: | The proposed benchmark measures task execution, complex reasoning, and noise robustness of agents by including 5 subsets, which set multiple evaluation dimensions. |
FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systems (2026.findings-acl)
Copied to clipboard
WenHao Wang, Haoting Shi, Mengying Yuan, Yiquan Lin, Panrong Tong, Hanzhang Zhou, Guangyi Liu, Pengxiang Zhao, Yue Wang, Siheng Chen
| Challenge: | a lack of benchmarks capture real-world, cross-platform heterogeneity in GUI training . traditional methods to train GUI agents rely on centralized data collection and manual labeling . |
| Approach: | They propose a benchmark for developing and evaluating federated GUI agents across mobile, web and desktop platforms. |
| Outcome: | The proposed benchmarks show that cross-platform collaboration improves performance and identify platform and OS as the most influential factors. |
FedMABench: Benchmarking Mobile GUI Agents on Decentralized Heterogeneous User Data (2025.emnlp-main)
Copied to clipboard
| Challenge: | Mobile GUI agents have attracted tremendous research participation recently. traditional approaches to mobile agent training rely on centralized data collection. |
| Approach: | They propose a benchmark for federated training and evaluation of mobile GUI agents . they find that federation algorithms consistently outperform local training . |
| Outcome: | The first benchmark for federated training and evaluation of mobile GUI agents is released . it features 6 datasets with 30+ subsets, 8 federation algorithms, 10+ base models, and over 800 apps across 5 categories . |
DAC-Bench: A Decision-Aware Benchmark for Compositional Mobile GUI Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on short, linear workflows and step-level accuracy, highlighting performance degradations. |
| Approach: | They propose a decision-aware benchmark with compositional tasks comprising 830 episodes and 11,345 action steps across 35 applications on Android and iOS. |
| Outcome: | The proposed benchmarks show performance degradation and branch correctness issues in 7 different GUI agents. |
MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains (2025.findings-naacl)
Copied to clipboard
Guoli Yin, Haoping Bai, Shuang Ma, Feng Nan, Yanchao Sun, Zhaoyang Xu, Shen Ma, Jiarui Lu, Xiang Kong, Aonan Zhang, Dian Ang Yap, Yizhe Zhang, Karsten Ahnert, Vik Kamath, Mathias Berglund, Dominic Walsh, Tobias Gindele, Juergen Wiest, Zhengfeng Lai, Xiaoming Simon Wang, Jiulong Shan, Meng Cao, Ruoming Pang, Zirui Wang
| Challenge: | Existing benchmarks focus on specific application scenarios, emphasizing task completion but failing to dissect the underlying skills that drive these outcomes. |
| Approach: | They propose a Massive Multitask Agent Understanding benchmark that evaluates LLMs across five domains and offline tasks. |
| Outcome: | The Massive Multitask Agent Understanding (MMAU) benchmark evaluates models across five domains including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. |
WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing GUI agents perform poorly on multi-application tasks, stalling at early sub-goals. |
| Approach: | They propose to assess GUI Agents on complex multi-step tasks that mirror real-world professions. |
| Outcome: | The proposed benchmark contains 181 tasks with an average of 5.0 sub-goals across 17 common desktop applications, of which 78% are inherently multi-application. |