Android in the Zoo: Chain-of-Action-Thought for GUI Agents (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on large language models (LLMs) focus on the semantics of smartphone operations. |
| Approach: | They propose a large language model (LLM) which predicts a sequence of actions of API by analyzing past actions and visual observations. |
| Outcome: | The proposed model improves the prediction of actions on a zero-shot Android-In-The-Zoo dataset compared to previous models . |
Similar Papers
You Only Look at Screens: Multimodal Chain-of-Action Agents (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to creating autonomous graphical user interfaces rely on external tools and application-specific APIs to interpret the environment. |
| Approach: | They propose a multimodal solution that directly interacts with the user interface without environment parsing. |
| Outcome: | The proposed solution bypasses environment parsing and reliance on application-dependent APIs. |
CoAT: Chain-of-Associated-Thoughts Framework for Enhancing Large Language Models Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | OpenAI-o1 enables ‘slow thinking’ because it is closer to the human thought process . |
| Approach: | They propose a new framework that integrates the Monte Carlo Tree Search algorithm and a dynamic mechanism for integrating new key information, termed ‘associative memory’. |
| Outcome: | The proposed framework improves performance on open-source multi-hop reasoning datasets and more than 15% gain on proprietary CRB dataset. |
CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation (2024.findings-acl)
Copied to clipboard
| Challenge: | Current vital challenges for autonomous agents lie in two aspects: dependence on strong (M)LLMs and insufficient GUI environment modeling. |
| Approach: | They propose a comprehensive cognitive LLM agent with two novel approaches to improve GUI automation performance. |
| Outcome: | The proposed agent achieves state-of-the-art performance on AITW and META-GUI benchmarks. |
Does Chain-of-Thought Reasoning Help Mobile GUI Agents? An Empirical Study (2026.findings-acl)
Copied to clipboard
| Challenge: | Reasoning capabilities have improved vision-language models in domains like math, coding, and visual question-answering, but their impact on real-world applications remains unclear. |
| Approach: | They evaluate six pairs of VLMs by comparing their base and reasoning-enhanced versions across static and interactive benchmarks. |
| Outcome: | The reasoning-enhanced models perform better on static and interactive benchmarks than non-reasoning models. |
AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Driving (2025.findings-emnlp)
Copied to clipboard
Kangan Qian, Sicong Jiang, Yang Zhong, Ziang Luo, Zilin Huang, Tianze Zhu, Kun Jiang, Mengmeng Yang, Zheng Fu, Jinyu Miao, Yining Shi, He Zhe Lim, Li Liu, Tianbao Zhou, Hongyi Wang, Huang Yu, Yifei Hu, Guang Li, Guang Chen, Hao Ye, Lijun Sun, Diange Yang
| Challenge: | Vision-Language Models struggle with hallucinations, inefficient reasoning, and limited real-world validation hinders accurate perception and robust step-by-step reasoning. |
| Approach: | AgentThink integrates Chain-of-Thought reasoning with dynamic, agent-style tool invocation for autonomous driving tasks. |
| Outcome: | Experiments on the DriveLMM-o1 benchmark show AgentThink significantly boosts overall reasoning scores by 53.91% and enhances answer accuracy by 33.54% . |
D-Artemis: A Deliberative Cognitive Framework for Mobile GUI Multi-Agents (2026.findings-acl)
Copied to clipboard
Hongze Mi, Yibo Feng, WenJie Lu, Yuqi Wang, Jinyuan Li, Song Cao, He Cui, Tengfei Tian, Xuelin Zhang, Haotian Luo, Di Sun, Jun Fang, Hua Chai, Naiqiang Tan, Gang Pan
| Challenge: | Graphical User Interface (GUI) agents aim to automate a wide spectrum of human tasks by emulating user interaction. |
| Approach: | They propose a deliberative framework that leverages a fine-grained tip retrieval mechanism to inform its decision-making process. |
| Outcome: | The proposed framework achieves SOTA among open-source general models on AndroidWorld and ScreenSpot-V2 . it leverages a fine-grained, app-specific tip retrieval mechanism to inform its decision-making process . |
GoT: Effective Graph-of-Thought Reasoning in Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have been advancing at an unprecedented pace. |
| Approach: | They propose a graph-based approach which models human thought processes as a chain and as 'graphs' by representing thought units as nodes and connections between them as edges, they capture the non-sequential nature of human thinking and allows for a more realistic modeling of thought processes. |
| Outcome: | The proposed model improves on a text-only reasoning task and a multimodal reasoning task. |
WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollback (2025.findings-emnlp)
Copied to clipboard
Minda Hu, Tianqing Fang, Jianshu Zhang, Jun-Yu Ma, Zhisong Zhang, Jingyan Zhou, Hongming Zhang, Haitao Mi, Dong Yu, Irwin King
| Challenge: | Web agents powered by Large Language Models lack the ability to perform in uncertain web environments. |
| Approach: | They propose to reconstruct web agents' reasoning skills into chain-of-thought rationales by fine-tuning their LLM backbone into a web-based model. |
| Outcome: | The proposed approach significantly improves the agent self-improving benchmark OpenWebVoyager, demonstrating that it can be used to improve the agent's reasoning skills. |
GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work of GUI action grounding fine-tunes data from pre-trained MLLMs, but data is limited to specific GUI environments. |
| Approach: | They propose to use a GUI-based agent to collect environment-specific data and fine-tune GUI grounding models with the collected data. |
| Outcome: | The proposed model can be extended to other GUI environments to improve performance. |
Nebula: A discourse aware Minecraft Builder (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has shown that at least some context is needed to understand and carry out conversationally given instructions. |
| Approach: | They propose to incorporate prior discourse and nonlinguistic contexts of a conversation situated in a nonlinguistic environment into an LLM model to improve the "language to action" component of collaborative tasks. |
| Outcome: | The proposed model doubles the baseline on the task of Jayannavar et al. (2020) and can construct shapes and understand location descriptions using a synthetic dataset. |