Understanding GUI Agent Localization Biases through Logit Sharpness (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal large language models often exhibit hallucinations that compromise reliability . despite promising performance, these models often display systematic localization errors . |
| Approach: | They propose a framework that categorizes model predictions into four distinct types . they propose metric that evaluates alignment between semantic continuity and logits distribution . |
| Outcome: | The proposed framework categorizes model predictions into four different types . it reveals nuanced failure modes beyond traditional accuracy metrics . |
Similar Papers
R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding (2025.findings-acl)
Copied to clipboard
Joonhyung Park, Peng Tang, Sagnik Das, Srikar Appalaraju, Kunwar Yashraj Singh, R. Manmatha, Shabnam Ghadar
| Challenge: | Existing vision-only GUI agents ground elements from large and cluttered screenshots, requiring them to process substantial irrelevant information that compromises their accuracy. |
| Approach: | They propose a visual agent model for GUI automation that leverages zoomed-in region proposals for precise element localization. |
| Outcome: | The proposed approach improves state-of-the-art grounding accuracy by 13% across diverse GUI platforms on the GUI grounding benchmarks ScreenSpot and AgentStudio. |
Can MLLMs Find Their Way in a City? Exploring Emergent Navigation from Web-Scale Knowledge (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks for multimodal large language models (MLLMs) are language-centric or heavily reliant on simulated environments, rarely probing the nuanced, knowledge-intensive reasoning essential for practical, real-world scenarios. |
| Approach: | They propose a task of Sparsely Grounded Visual Navigation to evaluate MLLM-driven agents in city navigation in four diverse global cities. |
| Outcome: | The proposed benchmark encompassing four diverse global cities evaluates agents' decision-making abilities in city navigation. |
Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractions (2025.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that multimodal GUI agents are susceptible to environmental distractions. |
| Approach: | They propose a scenario where both user and agent are benign and environment is not malicious . they implement an adversarial environment injection and analyze the approach to improve faithfulness . |
| Outcome: | The proposed approach improves faithfulness of multimodal large language model agents in a graphical user interface environment. |
Do GUI Grounders Truly Understand UI Elements? (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing grounding models and benchmarks are skewed toward web and mobile environments, neglecting desktop interfaces (especially windows). |
| Approach: | They propose a GUI Grounding Sensitivity Benchmark to assess UI grounding sensitivity to multiple descriptions of the same UI element. |
| Outcome: | The proposed model generates multiple valid instructions per UI element and develops nuanced validation methods to validate them. |
VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Multimodal Large Language Models have demonstrated remarkable capabilities across vision-language tasks, but their performance as embodied agents needs further exploration. |
| Approach: | They propose a framework to evaluate multimodal large language models as zero-shot agents . they find that enhancing prevalent agents with Chain-of-Thought reasoning and self-reflection leads to an unexpected performance decrease. |
| Outcome: | The proposed framework enables comparisons and component-level ablations across diverse MLLM architectures, agent designs, and navigation tasks. |
WinSpot: GUI Grounding Benchmark with Multimodal Large Language Models (2025.acl-short)
Copied to clipboard
| Challenge: | Existing GUI grounding data focuses on web-based elements, leaving a gap in real-world GUI interaction data for non-web applications. |
| Approach: | They propose a framework that leverages Large Language Models to generate large-scale GUI grounding data. |
| Outcome: | The framework validates and refines 5,000 GUI coordinate-instruction pairs and provides high-quality data for training and evaluating visual GUI agents. |
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection (2026.eacl-long)
Copied to clipboard
Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, Fei Wu
| Challenge: | Existing GUI Agents face challenges in multi-step reasoning and reliance on textual annotations, limiting their effectiveness. |
| Approach: | They propose an MLLM-based GUI Agent with a two-stage supervised fine-tuning pipeline that enhances GUI understanding and grounding. |
| Outcome: | InfiGUIAgent achieves competitive performance on several GUI benchmarks, highlighting the impact of native reasoning skills in enhancing GUI interaction for automation tasks. |
GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work of GUI action grounding fine-tunes data from pre-trained MLLMs, but data is limited to specific GUI environments. |
| Approach: | They propose to use a GUI-based agent to collect environment-specific data and fine-tune GUI grounding models with the collected data. |
| Outcome: | The proposed model can be extended to other GUI environments to improve performance. |
OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agents (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing efforts to build GUI agents focused on the autonomous mode have failed to address the problem of over-execution. |
| Approach: | They propose an adaptive GUI agent that predicts confidence levels at each interaction step and elicits adaptive interaction. |
| Outcome: | The proposed GUI agent outperforms existing models on a complex dataset and on established benchmarks. |
Protecting multimodal large language models against misleading visualizations (2026.acl-long)
Copied to clipboard
| Challenge: | MLLMs are robust to misleading visualizations, i.e., charts that distort the underlying data, leading readers to draw inaccurate conclusions. |
| Approach: | They propose to use table-based QA and redrawing the visualization to improve QA performance on misleading visualizations. |
| Outcome: | The proposed methods improve MLLM question-answering accuracy on misleading visualizations without compromising accuracy on non-misleading ones. |