PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | extending grouping-based methods to agentic reasoning presents unique challenges . frequent environment interactions and tool invocations render intra-group advantage estimation unstable . |
| Approach: | They propose a grouping-based method that uses a single round of rollouts to stabilize advantage estimation. |
| Outcome: | a new RL framework outperforms grouping-based methods in retrieval tasks and advanced mathematical reasoning benchmarks. |
Similar Papers
Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Reinforcement learning (RL) has re-emerged as a natural approach for training interactive LLM agents in real-world environments. |
| Approach: | They propose a variant that operates on a turn-level MDP formulation, instead of the commonly used token-level one. |
| Outcome: | The proposed method is more robust than the widely used GRPO algorithm and more efficient than token-level MDPs. |
LEPO: Latent Reasoning Policy Optimization for Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing latent reasoning methods that use chain of thought (CoT) are limited to selecting one discrete token at each reasoning step, which potentially induces information loss. |
| Approach: | They propose a framework that injects controllable stochasticity into latent reasoning via Gumbel-Softmax, restoring LLMs' exploratory capacity and enhancing their compatibility with Reinforcement Learning (RL). |
| Outcome: | The proposed framework preserves richer information for more comprehensive reasoning and is compatible with Reinforcement Learning (RL). |
BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards (2026.findings-acl)
Copied to clipboard
| Challenge: | Critic-free reinforcement learning with verifiable rewards (RLVR) is a practical paradigm for aligning Large Language Models. |
| Approach: | They propose a framework that stabilizes advantage estimation by combining prompt-local on-policy statistics with semantic-cluster-conditioned historical moments. |
| Outcome: | Experiments show that RLVR improves training stability and performance compared to critic-based methods . compared with other approaches, RL VR improves in cold-start regimes with binary verifiers . |
Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization (2026.acl-long)
Copied to clipboard
Yifeng Ding, Hung Le, Songyang Han, Kangrui Ruan, Zhenghui Jin, Varun Kumar, Zijian Wang, Anoop Deoras
| Challenge: | Current reinforcement learning methods suffer from coarse-grained, trajectory-level rewards that provide insufficient learning signals for complex multi-turn interactions, leading to training stagnation. |
| Approach: | They propose a novel RL algorithm for training large language models for multi-turn tool-integrated reasoning (TIR) that incorporates three innovations: turn-level reward assignment that provides fine-grained feedback for individual turns, return-based advantage estimation where normalized discounted returns are calculated as advantages, and self-supervised reward shaping that exploits self-supervision signals from generated code to densify sparse binary outcome-based rewards. |
| Outcome: | The proposed algorithm outperforms GRPO by 3.0% across diverse math reasoning benchmarks and improves grepo by 3.9% on commonsense reasoning and program synthesis tasks. |
AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin (2026.acl-long)
Copied to clipboard
| Challenge: | Reinforcement learning (RL) has emerged as an effective approach for enhancing the reasoning capabilities of large language models. |
| Approach: | They propose an algorithm that optimizes cross-entropy loss using advantages enhanced through a margin-based estimation scheme. |
| Outcome: | Experimental results show that AAPO improves group relative advantage estimation compared to other methods. |
Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing reinforcement learning methods rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions. |
| Approach: | They propose a process reward model that rewards correct steps only when they detect errors . they propose VPPO, which rewards the correct prefix and an erroneous suffix . |
| Outcome: | a new approach outperforms sparse-reward RL and prior PRM-guided baselines on Pass@1 and Pass@K . a process reward model (PRM) outperformed sparser-rebound RL on multiple reasoning benchmarks . |
Think Outside the Policy: In-Context Steered Policy Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing Reinforcement Learning from Verifiable Rewards (RLVR) methods exhibit limited exploration due to reliance on on-policy rollouts which are limited to the current policy’s distribution, resulting in narrow trajectory diversity. |
| Approach: | They propose a framework that leverages the in-context learning capability of Large Reasoning Models to provide expert guidance using existing datasets. |
| Outcome: | The proposed framework improves RLVR performance and training stability on mathematical reasoning benchmarks. |
VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training (2026.acl-long)
Copied to clipboard
Dingwei Zhu, Shihan Dou, Zhiheng Xi, Senjie Jin, Guoqiang Zhang, Jiazheng Zhang, Junjie Ye, Mingxu Chai, Enyu Zhou, Ming Zhang, Yuhui Wang, Caishuang Huang, Chenhao Huang, Yunke Zhang, Yuran Wang, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang
| Challenge: | Reinforcement Learning (RL) in real-world environments often suffers from ambiguous or incomplete supervision. |
| Approach: | They propose a framework that enhances value modeling for robust RL in LLM post-training by integrating auxiliary losses guided by entropy and perplexity from a frozen language model and variational information bottleneck. |
| Outcome: | The proposed framework outperforms baselines on multi-turn dialogue, math reasoning, and science QA with rule-based and model-based rewards. |
DRA-GRPO: Your GRPO Needs to Know Diverse Reasoning Paths for Mathematical Reasoning (2026.findings-acl)
Copied to clipboard
Xiwen Chen, Wenhui Zhu, Peijie Qiu, Xuanzhao Dong, Hao Wang, Haiyu Wu, Huayu Li, Aris Sotiras, Yalin Wang, Abolfazl Razi
| Challenge: | Existing methods for group-relative policy optimization rely on scalar correctness rewards that are often non-injective with respect to semantic content. |
| Approach: | They propose a framework that calibrates the reward signal using the semantic density of sampled groups. |
| Outcome: | The proposed framework outperforms strong baselines on five math benchmarks with 7,000 samples and 55 cost. |
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent preference-based fine-tuning methods have limited exploration in offline training . previous methods have been limited by the lack of exploration inherent in offline learning . |
| Approach: | They propose a method that normalizes rewards across a group of completed tasks to mitigate social bias in Large Language Models. |
| Outcome: | The proposed approach outperforms DPO and PPO in multiple benchmarks . it can overcome limitations of previous preference-based methods . |