D-Artemis: A Deliberative Cognitive Framework for Mobile GUI Multi-Agents (2026.findings-acl)
Copied to clipboard
Hongze Mi, Yibo Feng, WenJie Lu, Yuqi Wang, Jinyuan Li, Song Cao, He Cui, Tengfei Tian, Xuelin Zhang, Haotian Luo, Di Sun, Jun Fang, Hua Chai, Naiqiang Tan, Gang Pan
| Challenge: | Graphical User Interface (GUI) agents aim to automate a wide spectrum of human tasks by emulating user interaction. |
| Approach: | They propose a deliberative framework that leverages a fine-grained tip retrieval mechanism to inform its decision-making process. |
| Outcome: | The proposed framework achieves SOTA among open-source general models on AndroidWorld and ScreenSpot-V2 . it leverages a fine-grained, app-specific tip retrieval mechanism to inform its decision-making process . |
Similar Papers
KAPA: A Deliberative Agent Framework with Tree-Structured Knowledge Base for Multi-Domain User Intent Understanding (2025.findings-acl)
Copied to clipboard
Jiakai Tang, Shiqi Shen, ZhipengWang ZhipengWang, Gong Zhi, Xueyang Feng, Zexu Sun, Haoran Tan, Xu Chen
| Challenge: | Existing studies on the use of LLMs for estimating user intents are either too far from real human thought processes or require labeled samples. |
| Approach: | They propose a deliberative agent framework that leverages human thought process to build high-level domain knowledge and a tree-structured knowledge base to store refined experience and data. |
| Outcome: | The proposed framework is able to build high-level domain knowledge and efficiently store it across multiple steps. |
You Only Look at Screens: Multimodal Chain-of-Action Agents (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to creating autonomous graphical user interfaces rely on external tools and application-specific APIs to interpret the environment. |
| Approach: | They propose a multimodal solution that directly interacts with the user interface without environment parsing. |
| Outcome: | The proposed solution bypasses environment parsing and reliance on application-dependent APIs. |
Dynamic Planning for LLM-based Graphical User Interface Automation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to planning for GUI tasks are limited due to long historical dialogues. |
| Approach: | They propose a novel approach to dynamic planning based on environmental feedback and execution history to guide action prediction in GUI tasks. |
| Outcome: | The proposed approach surpasses the strong GPT-4V baseline by +12.7% in accuracy. |
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection (2026.eacl-long)
Copied to clipboard
Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, Fei Wu
| Challenge: | Existing GUI Agents face challenges in multi-step reasoning and reliance on textual annotations, limiting their effectiveness. |
| Approach: | They propose an MLLM-based GUI Agent with a two-stage supervised fine-tuning pipeline that enhances GUI understanding and grounding. |
| Outcome: | InfiGUIAgent achieves competitive performance on several GUI benchmarks, highlighting the impact of native reasoning skills in enhancing GUI interaction for automation tasks. |
Ponder & Press: Advancing Visual GUI Agent towards General Computer Control (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing multimodal large language models (MLLMs) lack visual inputs to ground objects, limiting flexibility across diverse software environments and platforms. |
| Approach: | They propose a divide-and-conquer framework for general computer control that uses only visual inputs to create a purely human-like interaction paradigm. |
| Outcome: | The proposed framework outperforms existing models by +22.5% on the ScreenSpot GUI grounding benchmark. |
A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation (2026.findings-acl)
Copied to clipboard
Yuxiang Chai, Shunye Tang, Han Xiao, Weifeng Lin, Hanhao Li, Jiayu Zhang, Liang Liu, Pengxiang Zhao, Guangyi Liu, Guozhi Wang, Shuai Ren, Rongduo Han, Haining Zhang, Siyuan Huang, Hongsheng Li
| Challenge: | Existing evaluation methods for mobile GUI agents rely on static frame assessments or offline static apps. |
| Approach: | They propose an evaluation system that leverages large language models as reward models to verify task completion and process achievement. |
| Outcome: | The proposed system addresses the limitations of traditional function based evaluation methods on online dynamic apps. |
ARGSBASE: A Multi-Agent Interface for Structured Human–AI Deliberation (2026.eacl-demo)
Copied to clipboard
| Challenge: | a new deliberation interface enables users to engage with multiple large language models (LLMs) ArgsBase exemplifies hybrid argumentation and supports epistemically responsible human–AI collaboration. |
| Approach: | They propose a deliberation interface that enables users to engage with multiple large language models coordinated by a moderator agent. |
| Outcome: | The proposed system exemplifies hybrid argumentation and aligns with recent calls for "reasonable parrots" the user study shows that the tool is easy to use, perspective-enhancing, and promising for research . |
NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Current research faces an "Evaluation-Realism Dilemma" due to unstable MLLM judges or manual verification. |
| Approach: | They propose a verifiable evaluation dataset grounded in real-world human GUI intents. |
| Outcome: | The proposed framework outperforms the state-of-the-art framework in achieving a weighted pathway success rate of 45.6% while reducing token consumption and execution time by 76%. |
INREACT: An Inspire-Then-Reinforce Training Framework For Multimodal GUI Agent (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multimodal large language models struggle with precise localization of small elements. |
| Approach: | They propose a multimodal GUI agent framework that unifies observing, thinking, and acting for precise and interpretable decision-making. |
| Outcome: | The proposed framework unifies observing, thinking, and acting for precise and interpretable decision-making. |
GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent (2025.acl-long)
Copied to clipboard
| Challenge: | GUI automation is a key challenge in dynamic environments. |
| Approach: | They propose a training-free GUI agent that integrates two mechanisms to explore trajectories in GUIs. |
| Outcome: | The proposed GUI-explorer shows significant improvements over existing agents. |