| Challenge: | Existing GUI agents for real desktop environments require large amounts of high-quality interaction data, but collecting human demonstrations is expensive. |
| Approach: | They propose a framework that bootstraps scalable desktop supervision from seed demonstrations. |
| Outcome: | Experiments on standard desktop benchmarks show that the framework improves on zero-shot agents and representative synthesis baselines. |
Similar Papers
OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis (2025.acl-long)
Copied to clipboard
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, Zhiyong Wu
| Challenge: | Graphical User Interface (GUI) agents powered by Vision-Language Models (VLMs) have demonstrated human-like computer control capability. |
| Approach: | They propose a GUI data synthesis pipeline that reverse engineers GUI trajectory construction process by executing pre-defined tasks. |
| Outcome: | The proposed GUI data synthesis pipeline overcomes the bottlenecks of previous methods that rely on pre-defined tasks and limited data diversity. |
GTA: Generating Long-horizon Tasks for Web Agents at Scale (2026.acl-long)
Copied to clipboard
Tenghao Huang, Kung-Hsiang Huang, Prafulla Kumar Choubey, Yilun Zhou, Muhao Chen, Jonathan May, Chien-Sheng Wu
| Challenge: | Existing benchmarks provide only coarse start–goal annotations without intermediate trajectories . Existing frameworks provide no supervision over the agent's latent decision process . |
| Approach: | They propose a framework that integrates crawling, retrieval-based seeding, in-context generation and automated quality control to produce realistic tasks paired with executable trajectories. |
| Outcome: | The proposed framework decouples crawling from generation for greater efficiency and ensures dense supervision through deterministic replays and systematic validation. |
GUI Agents: A Survey (2025.findings-acl)
Copied to clipboard
Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi, Hongjie Chen, Viet Dac Lai, Zhouhang Xie, Sungchul Kim, Ruiyi Zhang, Tong Yu, Mehrab Tanjim, Nesreen K. Ahmed, Puneet Mathur, Seunghyun Yoon, Lina Yao, Branislav Kveton, Jihyung Kil, Thien Huu Nguyen, Trung Bui, Tianyi Zhou, Ryan A. Rossi, Franck Dernoncourt
| Challenge: | Large Foundation Models (LFMs) have transformed the landscape of AI research and day-to-day life. |
| Approach: | They propose a framework that delineates GUI agents' perception, reasoning, planning, and acting capabilities. |
| Outcome: | The proposed framework delineates their perception, reasoning, planning, and acting capabilities. |
From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation (2026.acl-long)
Copied to clipboard
| Challenge: | Vision-language models are increasingly deployed as computer-use agents that operate desktops and browsers. |
| Approach: | They propose a method that turns static expert traces into policy-aligned guidance . they propose RLVR with a per-task, dynamically updated cache to decompose planning and execution . |
| Outcome: | The proposed model improves UITARS1.5-7B success from 22.87% to 32.13% on OSWorld-Verified and raises a held-out split from 5.74% to 10.30% on MMBench-GUI and Online-Mind2Web. |
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (2024.acl-long)
Copied to clipboard
| Challenge: | Existing GUI agents interact with the environment through extracted structured data, which can be notably lengthy (e.g., HTML) and occasionally inaccessible (e-book). |
| Approach: | They propose to enhance SeeClick with GUI grounding pre-training and devise a method to automate curation of GUI ground data. |
| Outcome: | The proposed agent improves ScreenSpot, the first realistic GUI grounding benchmark that encompasses mobile, desktop, and web environments. |
NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Current research faces an "Evaluation-Realism Dilemma" due to unstable MLLM judges or manual verification. |
| Approach: | They propose a verifiable evaluation dataset grounded in real-world human GUI intents. |
| Outcome: | The proposed framework outperforms the state-of-the-art framework in achieving a weighted pathway success rate of 45.6% while reducing token consumption and execution time by 76%. |
Do GUI Grounders Truly Understand UI Elements? (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing grounding models and benchmarks are skewed toward web and mobile environments, neglecting desktop interfaces (especially windows). |
| Approach: | They propose a GUI Grounding Sensitivity Benchmark to assess UI grounding sensitivity to multiple descriptions of the same UI element. |
| Outcome: | The proposed model generates multiple valid instructions per UI element and develops nuanced validation methods to validate them. |
AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories (2024.findings-emnlp)
Copied to clipboard
Yifan Song, Weimin Xiong, Xiutian Zhao, Dawei Zhu, Wenhao Wu, Ke Wang, Cheng Li, Wei Peng, Sujian Li
| Challenge: | Existing studies focus on specialized agents designed for particular tasks. |
| Approach: | They propose to scale annotated interaction trajectories and fine-tune LLMs on AgentBank to get a series of agent models, Samoyed. |
| Outcome: | The proposed model can scale to get generalized agent capabilities. |
Aria-UI: Visual Grounding for GUI Instructions (2025.findings-acl)
Copied to clipboard
| Challenge: | Using a multimodal model, GUI agents can ground from language instructions to target elements . relying on HTML or AXTree inputs is a challenge for GUI agents . |
| Approach: | They propose a large multimodal model specifically designed for GUI grounding that adopts a pure vision approach instead of auxiliary inputs. |
| Outcome: | The proposed model outperforms vision-only and AXTree-reliant models on offline and online agents. |
ReCoQA: A Benchmark for Tool-Augmented and Multi-Step Reasoning in Real Estate Question and Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Real estate agents are labor-intensive, difficult to scale, and prone to interest-driven bias. |
| Approach: | They propose a large-scale benchmark of 29,270 real-estate instances with machine-verifiable supervision for intermediate steps . they propose 'hIRE-Agent' framework that integrates heterogeneous evidence into an understand–plan–execute architecture as a strong baseline . |
| Outcome: | Experiments show that HIRE-Agent integrates heterogeneous evidence . the framework is able to integrate a front-end parser, planning Supervisor, and execution Specialists . |