Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding (2026.acl-long)
Copied to clipboard
| Challenge: | Graphical User Interface (GUI) grounding requires mapping natural language instructions to precise pixel coordinates due to visually homogeneous elements and dense layouts. |
| Approach: | They propose to replace static consistency strategies with a learnable selection mechanism that selects the optimal target by critiquing its own proposals rendered on the screenshot. |
| Outcome: | The proposed model significantly improves both grounding and critiquing capabilities over 6 benchmarks. |
Similar Papers
No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning (2026.acl-long)
Copied to clipboard
Zhicong Li, Lingjie Jiang, Yulan Hu, Xingchen Zeng, Yixia Li, Xiangwen Zhang, Guanhua Chen, Zheng Pan, Xin Li, Yong Liu
| Challenge: | Current methods for training Large Language Model agents rely on static or offline critic models, which fail to adapt as the policy evolves. |
| Approach: | They propose a framework that integrates a critique and a policy to optimize the policy and critic through a synchronized co-evolutionary loop. |
| Outcome: | The proposed framework yields more stable training and higher long-horizon task success across open-world environments. |
CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing search agent pipelines rely on sparse outcome rewards, leading to inefficient exploration and unstable training. |
| Approach: | They propose a tool-integrated reasoning framework that provides turn-level feedback via a retrospective critic mechanism. |
| Outcome: | The proposed framework outperforms baselines in multi-hop reasoning benchmarks and achieves faster convergence and training stability. |
DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | DiMo-GUI is a training-free framework for GUI grounding that splits input into textual elements and iconic elements, allowing the model to reason over each modality independently using general-purpose vision-language models. |
| Approach: | They propose a training-free framework for GUI grounding that leverages two core strategies: dynamic visual grounding and modality-aware optimization. |
| Outcome: | The proposed framework splits the input into textual elements and iconic elements, allowing the model to reason over each modality independently using general-purpose vision-language models. |
Do GUI Grounders Truly Understand UI Elements? (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing grounding models and benchmarks are skewed toward web and mobile environments, neglecting desktop interfaces (especially windows). |
| Approach: | They propose a GUI Grounding Sensitivity Benchmark to assess UI grounding sensitivity to multiple descriptions of the same UI element. |
| Outcome: | The proposed model generates multiple valid instructions per UI element and develops nuanced validation methods to validate them. |
Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for grounding large language models suffer from inefficient querying . Existing approaches that rely on physical verification or self-reflection suffer from excessive querying. |
| Approach: | They propose a framework that introduces Reinforced Advantage feedback for efficient self-refinement of plans. |
| Outcome: | The proposed framework surpasses baselines in success rate and significantly decreases interaction steps of agents and query rounds of LLMs. |
TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments (2025.findings-acl)
Copied to clipboard
Yuheng Lu, Qian Yu, Hongru Wang, Zeming Liu, Wei Su, Yanping Liu, Yuhang Guo, Maocheng Liang, Yunhong Wang, Haifeng Wang
| Challenge: | Existing GUI agents struggle to adapt to dynamic and interconnected nature of real-world digital environments, authors show . |
| Approach: | They propose a benchmark to evaluate the transferability of GUI agents across three key dimensions . transBench includes 15 app categories with diverse functionalities . |
| Outcome: | The proposed benchmark shows that existing GUI agents struggle to adapt to dynamic, interconnected environments. |
CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalization (2026.acl-long)
Copied to clipboard
Zhongyuan Peng, Yifan Yao, Kaijing Ma, Shuyue Guo, Yizhe Li, Yichi Zhang, Chenchen Zhang, Yifan Zhang, Zhouliang Yu, Luming Li, Minghao Liu, Yihang Xia, Jiawei Shen, Yuchen Wu, Yixin Cao, Zhaoxiang Zhang, Wenhao Huang, Jiaheng Liu, Ge Zhang
| Challenge: | Existing approaches to formalizing mathematical statements face limitations in accuracy, especially in the context of complex, highlevel problems that involve sophisticated mathematical reasoning. |
| Approach: | They propose a CriticLean framework that elevates the role of the critic from a passive validator to an active learning component and introduce a benchmark to measure models’ ability to distinguish semantically correct from incorrect formalizations. |
| Outcome: | The proposed framework outperforms open- and closed-source benchmarks and shows that it significantly outperformed existing models. |
Datasets and Recipes for Video Temporal Grounding via Reinforcement Learning (2025.emnlp-industry)
Copied to clipboard
Ruizhe Chen, Tianze Luo, Zhiting Fan, Heqing Zou, Zhaopeng Feng, Guiyang Xie, Hansheng Zhang, Zhuochen Wang, Zuozhu Liu, Zhang Huaijian
| Challenge: | Existing methods for video temporal grounding suffer from limited temporal awareness and poor generalization. |
| Approach: | They propose a two-stage training framework that integrates supervised fine-tuning with reinforcement learning to improve both the accuracy and robustness of VTG models. |
| Outcome: | The proposed training framework outperforms existing models on multiple benchmarks on open-domain and challenging scenarios. |
Leveraging Outline-Optimized Generative Interactions and Critique for Self-Refining Outlines with Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Logic-RL is a framework that transforms critique-guided outline refinement into a learnable policy through reinforcement learning. |
| Approach: | They propose a framework that transforms critique-guided outline refinement into a learnable policy through reinforcement learning. |
| Outcome: | The proposed framework improves on FreshWiki and WikiOutline . it can be iteratively applied, with improved quality continuing through three refinement rounds before diminishing returns. |
SEE: Strategic Exploration and Exploitation for Cohesive In-Context Prompt Optimization (2025.acl-long)
Copied to clipboard
Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley A. Malin, Sricharan Kumar
| Challenge: | Existing approaches separate the optimization of prompt instructions and in-context learning examples, leading to incohesive, suboptimal results. |
| Approach: | They propose a framework that refines both prompt instructions and in-context learning examples. |
| Outcome: | The proposed framework outperforms state-of-the-art prompt optimization methods on 35 benchmark tasks. |