| Challenge: | Prior studies modeled multimodal UI grounding in one round, but such an interaction is inherently iterative. |
| Approach: | They propose a task where a user and an agent collaborate on an interface screen . they use a dataset of 77,820 sequences of human user-agent interaction on mobile interfaces . |
| Outcome: | The proposed task improves the absolute task completion by 18% over the entire test set and 31% over the challenging split. |
Similar Papers
Do GUI Grounders Truly Understand UI Elements? (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing grounding models and benchmarks are skewed toward web and mobile environments, neglecting desktop interfaces (especially windows). |
| Approach: | They propose a GUI Grounding Sensitivity Benchmark to assess UI grounding sensitivity to multiple descriptions of the same UI element. |
| Outcome: | The proposed model generates multiple valid instructions per UI element and develops nuanced validation methods to validate them. |
You Only Look at Screens: Multimodal Chain-of-Action Agents (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to creating autonomous graphical user interfaces rely on external tools and application-specific APIs to interpret the environment. |
| Approach: | They propose a multimodal solution that directly interacts with the user interface without environment parsing. |
| Outcome: | The proposed solution bypasses environment parsing and reliance on application-dependent APIs. |
Aria-UI: Visual Grounding for GUI Instructions (2025.findings-acl)
Copied to clipboard
| Challenge: | Using a multimodal model, GUI agents can ground from language instructions to target elements . relying on HTML or AXTree inputs is a challenge for GUI agents . |
| Approach: | They propose a large multimodal model specifically designed for GUI grounding that adopts a pure vision approach instead of auxiliary inputs. |
| Outcome: | The proposed model outperforms vision-only and AXTree-reliant models on offline and online agents. |
GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work of GUI action grounding fine-tunes data from pre-trained MLLMs, but data is limited to specific GUI environments. |
| Approach: | They propose to use a GUI-based agent to collect environment-specific data and fine-tune GUI grounding models with the collected data. |
| Outcome: | The proposed model can be extended to other GUI environments to improve performance. |
Grounding Task Assistance with Multimodal Cues from a Single Demonstration (2025.findings-acl)
Copied to clipboard
Gabriel Herbert Sarch, Balasaravanan Thoravi Kumaravel, Sahithya Ravi, Vibhav Vineet, Andrew D Wilson
| Challenge: | RGB video often fails to capture fine-grained contextual cues such as intent, safety-critical environmental factors, and subtle preferences embedded in human behavior. |
| Approach: | They propose a framework that integrates eye gaze and speech cues to improve conversational agents for task assistance by integrating eye gaze with speech cuests. |
| Outcome: | The proposed framework captures fine-grained intent and user-specific cues, enabling richer contextual grounding for visual question answering. |
TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments (2025.findings-acl)
Copied to clipboard
Yuheng Lu, Qian Yu, Hongru Wang, Zeming Liu, Wei Su, Yanping Liu, Yuhang Guo, Maocheng Liang, Yunhong Wang, Haifeng Wang
| Challenge: | Existing GUI agents struggle to adapt to dynamic and interconnected nature of real-world digital environments, authors show . |
| Approach: | They propose a benchmark to evaluate the transferability of GUI agents across three key dimensions . transBench includes 15 app categories with diverse functionalities . |
| Outcome: | The proposed benchmark shows that existing GUI agents struggle to adapt to dynamic, interconnected environments. |
Achieving Common Ground in Multi-modal Dialogue (2020.acl-tutorials)
Copied to clipboard
| Challenge: | tutorial focuses on three main topic areas: grounding in human-human communication, dialogue systems and multi-modal interactive systems. |
| Approach: | This tutorial examines the use of computational dialogue research to design grounding modules and behaviors in cutting-edge systems. |
| Outcome: | This tutorial examines the results of recent research on grounding in human-human communication . it shows how these results lead to rich and challenging opportunities for doing grounding more flexible and powerful ways . |
WinSpot: GUI Grounding Benchmark with Multimodal Large Language Models (2025.acl-short)
Copied to clipboard
| Challenge: | Existing GUI grounding data focuses on web-based elements, leaving a gap in real-world GUI interaction data for non-web applications. |
| Approach: | They propose a framework that leverages Large Language Models to generate large-scale GUI grounding data. |
| Outcome: | The framework validates and refines 5,000 GUI coordinate-instruction pairs and provides high-quality data for training and evaluating visual GUI agents. |
MuG: A Multimodal Classification Benchmark on Game Data with Tabular, Textual, and Visual Fields (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multimodal classification systems use tabular, textual, and visual data to provide efficient and scalable services. |
| Approach: | They propose a multimodal classification benchmark MuG with eight datasets . they analyze label balance ratios, percentages of missing features, distributions of data within each modality . |
| Outcome: | The proposed benchmark is available on https://github.com/lujiaying/MUG-Bench . it includes eight datasets that allow researchers to evaluate and improve their models . |
DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboards (2026.findings-eacl)
Copied to clipboard
Aaryaman Kartha, Ahmed Masry, Mohammed Saidul Islam, Thinh Lang, Shadikur Rahman, Ridwan Mahbub, Mizanur Rahman, Mahir Ahmed, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty
| Challenge: | Existing question-answering benchmarks for data visualizations focus on static charts instead of interactive dashboards. |
| Approach: | They propose a benchmark to assess how vision-language GUI agents comprehend and interact with real-world dashboards. |
| Outcome: | The first benchmark explicitly designed to assess how vision-language GUI agents comprehend and interact with real-world dashboards. |