Experience-driven Multi-turn Reinforcement Learning for GUI Agents

Challenge:	GUI agents have demonstrated remarkable progress in automating complex user interface interactions . training such agents for long-horizon tasks remains challenging due to limited rewards and prohibitive costs.
Approach:	They propose a method that leverages expert trajectories as environment experiences for on-policy multi-turn training.
Outcome:	The proposed method achieves significant gains over the base model with 1K public trajectories as RL experiences . it achieves competitive performance against strong baselines such as UI-TARS-7B and GPT-4o .

Similar Papers

Overview

Challenge

Approach

Outcome

From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation (2026.acl-long)

Copied to clipboard

Zezhou Wang, Ziyun Zhang, Xiaoyi Zhang, Zhuzhong Qian, Yan Lu

Challenge:	Vision-language models are increasingly deployed as computer-use agents that operate desktops and browsers.
Approach:	They propose a method that turns static expert traces into policy-aligned guidance . they propose RLVR with a per-task, dynamically updated cache to decompose planning and execution .
Outcome:	The proposed model improves UITARS1.5-7B success from 22.87% to 32.13% on OSWorld-Verified and raises a held-out split from 5.74% to 10.30% on MMBench-GUI and Online-Mind2Web.

Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization (2026.acl-long)

Copied to clipboard

Yifeng Ding, Hung Le, Songyang Han, Kangrui Ruan, Zhenghui Jin, Varun Kumar, Zijian Wang, Anoop Deoras

Reasoning

Challenge:	Current reinforcement learning methods suffer from coarse-grained, trajectory-level rewards that provide insufficient learning signals for complex multi-turn interactions, leading to training stagnation.
Approach:	They propose a novel RL algorithm for training large language models for multi-turn tool-integrated reasoning (TIR) that incorporates three innovations: turn-level reward assignment that provides fine-grained feedback for individual turns, return-based advantage estimation where normalized discounted returns are calculated as advantages, and self-supervised reward shaping that exploits self-supervision signals from generated code to densify sparse binary outcome-based rewards.
Outcome:	The proposed algorithm outperforms GRPO by 3.0% across diverse math reasoning benchmarks and improves grepo by 3.9% on commonsense reasoning and program synthesis tasks.

STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimization (2026.findings-acl)

Copied to clipboard

Yuhan Chen, Yuxuan Liu, Long Zhang, Pengzhi Gao, Jian Luan, Wei Liu

Green & Sustainable NLP

Challenge:	Existing GRPO-based methods allocate sampling uniformly across tasks regardless of difficulty, propagate misleading learning signals and incur high sample-collection costs.
Approach:	They propose a framework that allocates sampling based on per-task success rates and performs fine-grained step-level optimization.
Outcome:	The proposed method improves sample efficiency and training stability over existing GRPO variants and three ablation variants on OSWorld and AndroidWorld.

SOLAR-RL: Semi-Online Long-horizon Assignment Reinforcement Learning (2026.findings-acl)

Copied to clipboard

Jichao Wang, Liuyang Bian, Yufeng Zhou, Han Xiao, Yue Pan, Guozhi Wang, Hao Wang, Zhaoxiong Wang, Yafei Wen, Xiaoxin Chen, Shuai Ren, Lingfang Zeng

Language Models

Challenge:	Existing approaches to training GUI agents on dynamic tasks are based on SFT or Behavior Cloning.
Approach:	They propose a framework that integrates global trajectory insights directly into offline learning . they reconstruct diverse rollout candidates from static data and detect first failure point .
Outcome:	The proposed framework improves long-horizon task completion rates and robustness compared to baselines.

LUMINA: Long-horizon Understanding for Multi-turn Interactive Agents (2026.findings-acl)

Copied to clipboard

Amin Rakhsha, Thomas Hehn, Pietro Mazzaglia, Fabio Valerio Massoli, Arash Behboodi, Tribhuvanesh Orekondy

Language Models

Challenge:	Large language models struggle on multi-turn, long-horizon agentic problems that require skills such as planning, state tracking, and long context processing.
Approach:	They propose an oracle counterfactual framework for multi-turn problems that asks: how would an agent perform if it could leverage an or acle to execute a specific skill?
Outcome:	The proposed framework allows for precise oracle interventions without confounding effects present in real-world benchmarks.

Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs (2026.findings-eacl)

Copied to clipboard

Junbo Li, Peng Zhou, Rui Meng, Meet P. Vadera, Lihong Li, Yang Li

Reasoning

Challenge:	Reinforcement learning (RL) has re-emerged as a natural approach for training interactive LLM agents in real-world environments.
Approach:	They propose a variant that operates on a turn-level MDP formulation, instead of the commonly used token-level one.
Outcome:	The proposed method is more robust than the widely used GRPO algorithm and more efficient than token-level MDPs.

Mobile-R1: Towards Interactive Capability for VLM-Based Mobile Agent via Systematic Training (2026.acl-long)

Copied to clipboard

Jihao Gu, Qihang Ai, Yingyao Wang, Pi Bu, Jingxuan Xing, Yue Cao, Zekun Zhu, Wei Jiang, Ziming Wang, Yingxiu Zhao, Ming-Liang Zhang, Jun Song, Yuning Jiang, Bo Zheng

Semantic Text Processing

Challenge:	Existing approaches to training agents for visual-language models trap them in local optima, hindering exploration and error correction with the environment.
Approach:	They propose a hierarchical training recipe that bridges atomic action execution and strategic task completion.
Outcome:	The proposed training recipe bridges atomic action execution and strategic task completion.

WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning (2025.emnlp-main)

Copied to clipboard

Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, Lihong Li

Challenge:	Existing work on reinforcement learning has focused on single-turn tasks such as solving math problems.
Approach:	They propose a framework that learns directly from online interactions by asynchronously generating diverse trajectories, guided by binary rewards depending on task success.
Outcome:	Experiments on the WebArena-Lite benchmark show that the framework outperforms state-of-the-art methods and strong proprietary models.

UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization (2026.acl-long)

Copied to clipboard

Zhengxi Lu, Fei Tang, Guangyi Liu, Jin Ma, Kaitao Song, Xu Tan, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

Information Retrieval

Challenge:	Experimental results show that UI-Copilot-7B achieves state-of-the-art performance on challenging MemGUI-Bench, outperforming strong 7B-scale GUI agents such as GUI-Owl-7B and UITARS-1.5-7B.
Approach:	They propose a collaborative framework where the GUI agent focuses on task execution while a lightweight copilot provides on-demand assistance for memory retrieval and numerical computation.
Outcome:	The proposed framework outperforms GUI-Owl-7B and UI-TARS-1.5-7B on MemGUI-Bench and delivers 17.1% improvement on AndroidWorld over the base Qwen model.

MGPO: Thinking with Images via Multi-Turn Grounding-Based Reinforcement Learning (2026.findings-acl)

Copied to clipboard

Xinyu Huang, Yuhao Dong, Weiwei Tian, Bo Li, Rui Feng, Ziwei Liu

Multimodality

Challenge:	State-of-the-art large multimodal models face challenges when processing high-resolution images, as these inputs are converted into enormous visual tokens, many of which are irrelevant to the downstream task.
Approach:	They propose a multi-turn grounding-based policy optimization framework that enables LMMs to iteratively focus on key visual regions by automatically cropping sub-images based on model-predicted grounding coordinates within a multiple-turn conversation framework.
Outcome:	The proposed framework improves on Qwen2.5-VL-7B with 21K samples and surpasses OpenAI’s o1 and GPT-4o models on the out-of-distribution (OOD) V* Bench.