Mapping Natural Language Instructions to Mobile UI Action Sequences (2020.acl-main)
Copied to clipboard
| Challenge: | a new problem of grounding natural language instructions to mobile UI actions is emerging . we use a Transformer to extract action phrase tuples from long-range natural language instruction . |
| Approach: | They propose a dataset that pairs English instructions with actions performed by people on a mobile UI emulator. |
| Outcome: | The proposed model achieves 70.59% accuracy on predicting complete ground-truth action sequences in PixelHelp. |
Similar Papers
From Perception to Reasoning: Enhancing Vision-Language Models for Mobile UI Understanding (2025.findings-acl)
Copied to clipboard
Settaluri Lakshmi Sravanthi, Ankit Mishra, Debjyoti Mondal, Subhadarshi Panda, Rituraj Singh, Pushpak Bhattacharyya
| Challenge: | Accurately grounding visual and textual elements within mobile user interfaces remains a challenge for Vision-Language Models (VLMs). |
| Approach: | They propose a mobile UI understanding model trained on a dataset specifically tailored for mobile screen understanding and grounding. |
| Outcome: | The proposed model achieves significant gains in accuracy across all perception tasks and on reasoning benchmarks. |
Aria-UI: Visual Grounding for GUI Instructions (2025.findings-acl)
Copied to clipboard
| Challenge: | Using a multimodal model, GUI agents can ground from language instructions to target elements . relying on HTML or AXTree inputs is a challenge for GUI agents . |
| Approach: | They propose a large multimodal model specifically designed for GUI grounding that adopts a pure vision approach instead of auxiliary inputs. |
| Outcome: | The proposed model outperforms vision-only and AXTree-reliant models on offline and online agents. |
UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis (2025.findings-acl)
Copied to clipboard
| Challenge: | Graphical User Interface (GUI) agents that utilize human-like vision perception capabilities are gaining a wider applicability compared to GUI metadata-based approaches. |
| Approach: | They propose a large-scale data synthesis pipeline for generating varying complex instruction datasets using GPT-4o instead of human annotators. |
| Outcome: | The proposed model achieves superior performance in GUI instruction grounding, demonstrating the advancements of proposed data synthesis pipeline. |
From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Grounding (2025.acl-long)
Copied to clipboard
| Challenge: | a pursuit of diverse, complex, and large-scale instruction data is crucial for automatically aligning large language models . authors: methods that generate synthetic instructions at scale suffer from limited grounding sources . attributed grounding is a technique that can be used to align language models with human . |
| Approach: | They synthesize 1 million instructions using attributed grounding and a bottom-up synthesis process that leverages web documents to generate a situation, then a meaningful instruction. |
| Outcome: | The proposed framework achieves leading performance on benchmarks and scales with more web corpora. |
Do GUI Grounders Truly Understand UI Elements? (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing grounding models and benchmarks are skewed toward web and mobile environments, neglecting desktop interfaces (especially windows). |
| Approach: | They propose a GUI Grounding Sensitivity Benchmark to assess UI grounding sensitivity to multiple descriptions of the same UI element. |
| Outcome: | The proposed model generates multiple valid instructions per UI element and develops nuanced validation methods to validate them. |
Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentions (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for visual grounding rely on the assumption that the given expression must be literal . this impedes the practical deployment of agents in real-world scenarios. |
| Approach: | They propose a visual grounding task that uses intention expressions to locate foreground entities . they build a large-scale IVG dataset with free-form intention expression to promote VG . |
| Outcome: | The proposed method is based on a large-scale intention-driven visual-language (V-L) dataset with free-form intention expressions. |
Lexi: Self-Supervised Learning of the UI Language (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models rely on UI metadata, which is often missing or not accessible. |
| Approach: | They propose a vision and language model that can learn generic visio-linguistic representations of UIs . they use a dataset to train Lexi, which is based on UI metadata . |
| Outcome: | The proposed model can handle unique features of UI screens, including text richness and context sensitivity. |
tagE: Enabling an Embodied Agent to Understand Human Instructions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems for natural language understanding (NLU) are limited due to the inherent ambiguity and incompleteness inherent in natural language. |
| Approach: | They propose a system to extract tasks from natural language instructions and map them to robots' established collection of skills. |
| Outcome: | The proposed system outperforms baseline models in the training and evaluation of a dataset featuring complex instructions. |
GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work of GUI action grounding fine-tunes data from pre-trained MLLMs, but data is limited to specific GUI environments. |
| Approach: | They propose to use a GUI-based agent to collect environment-specific data and fine-tune GUI grounding models with the collected data. |
| Outcome: | The proposed model can be extended to other GUI environments to improve performance. |
Language in a (Search) Box: Grounding Language Learning in Real-World Human-Machine Interaction (2021.naacl-main)
Copied to clipboard
| Challenge: | Scholarly work in this area uses toy worlds and synthetic linguistic data, but grounded language learning offers several practical and scientific advantages. |
| Approach: | They propose to model teacher-learner dynamics through natural interactions occurring between users and search engines. |
| Outcome: | The proposed model is better than non-grounded models on compositionality and zero-shot inference tasks. |