RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization (2026.acl-long)
Copied to clipboard
Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Yingwei MA, Jue Chen, Binhua Li, Zhi Jin, Fei Huang, Yongbin Li, Ge Li
| Challenge: | Reinforcement Learning with Verifiable Reward (RLVR) has significantly advanced the complex reasoning abilities of Large Language Models (LLMs). |
| Approach: | They propose a hybrid-policy optimization approach that synergizes internal exploitation with external data to achieve stronger reasoning capabilities. |
| Outcome: | The proposed approach achieves state-of-the-art performance on six math reasoning benchmarks and superior performance on out-of distribution reasoning tasks. |
Similar Papers
Does RLVR Extend Reasoning Boundaries? Investigating Capability Expansion in Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Recent studies suggest that RLVR amplifies behaviors inherent to the pre-training distribution rather than inducing new capabilities. |
| Approach: | They propose a framework for RLVR that extends the spatial reasoning boundary . they use a mapping framework where the difficulty is precisely regulated by path length and number of turns . |
| Outcome: | The proposed framework extends the spatial reasoning boundary on two real-world navigation benchmarks. |
Revisiting Entropy in Reinforcement Learning for Large Reasoning Models (2026.findings-acl)
Copied to clipboard
Renren Jin, Pengzhi Gao, Yuqi Ren, Zhuowen Han, Tongxuan Zhang, Wuwei Huang, Wei Liu, Jian Luan, Deyi Xiong
| Challenge: | Reinforcement learning with verifiable rewards (RLVR) has emerged as a paradigm for enhancing the reasoning capabilities of large language models. |
| Approach: | They propose a positive-advantage reweighting approach that regulates model entropy by adjusting the loss weights assigned to tokens with positive advantages during RLVR training. |
| Outcome: | The proposed approach regulates model entropy by adjusting loss weights assigned to tokens with positive advantages during RLVR training while maintaining competitive performance. |
Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domains (2026.acl-long)
Copied to clipboard
Zhonghang Yuan, Zhefan Wang, Fang Hu, Zihong Chen, Jinzhe Li, Gang Li, Jie Ying, Huanjun Kong, Songyang Zhang, Nanqing Dong
| Challenge: | Recent large language models (LLMs) have demonstrated remarkable progress in reasoning, but their applications on knowledge-intensive domains have not been explored due to the scarcity of high-quality verifiable data. |
| Approach: | They propose a framework that extends reinforcement learning with verifiable rewards (RLVR) to knowledge-intensive domains through automated verififiability data synthesis while enabling verification of the LLM's reasoning process. |
| Outcome: | Extensive experiments show that the proposed framework enhances the reasoning of large language models in knowledge-intensive domains without significantly compromising the model’s general capabilities. |
Crossing the Reward Bridge: Expanding Reinforcement Learning with Verifiable Rewards Across Diverse Domains (2026.acl-long)
Copied to clipboard
| Challenge: | Reinforcement learning with verifiable rewards (RLVR) has been effective on structured tasks, but its reliance on simple, rule-based verifiers creates a bottleneck. |
| Approach: | They propose a framework that uses a generative verifier to provide soft, probabilistic rewards. |
| Outcome: | The proposed framework outperforms existing models up to 10x their size and can be scalable and effective. |
How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization (2026.findings-acl)
Copied to clipboard
Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi, Chaowen Hu, Lu Pan, Ke Zeng, Xunliang Cai
| Challenge: | Existing methods for reinforcement learning with verifiable rewards are limited by the complexity of the problem and the complexity. |
| Approach: | They propose a theoretically-grounded dual-pronged optimization framework for reinforcement learning with verifiable rewards that compensates for gradient attenuation of high-confidence correct actions while utilizing entropy changes as computable indicators to stabilize excessive update magnitudes. |
| Outcome: | The proposed framework compensates for gradient attenuation of high-confidence correct actions while utilizing entropy changes as computable indicators to stabilize excessive update magnitudes. |
Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts (2026.acl-long)
Copied to clipboard
Sijia Luo, Xiaokang Zhang, Yuxuan Hu, Bohan Zhang, Ke Wang, Jinbo Su, Mengshu Sun, Lei Liang, Jing Zhang
| Challenge: | Existing methods for storing key-value caches during long-horizon rollouts cause performance collapses. |
| Approach: | They propose a new training paradigm that empowers stable RL training under sparse rollouts. |
| Outcome: | The proposed model reduces rollout overhead while maintaining the performance. |
From log 𝜋 to 𝜋: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight (2026.acl-long)
Copied to clipboard
Xiaoliang Fu, Jiaye Lin, Yangyi Fang, Chaowen Hu, Cong Qin, Zekai Shao, Binbin Zheng, Lu Pan, Ke Zeng
| Challenge: | Standard algorithms for Large Language Models (LLMs) enforce stability via "hard clipping" but relying on log-probability gradient yields divergent weights as probabilities vanish, destabilizing LLM training. |
| Approach: | They propose a decoupled gradient policy optimization that uses a decay mechanism to decouple the probability of a boundary token. |
| Outcome: | The proposed algorithm outperforms baselines on various mathematical benchmarks. |
Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models (2026.findings-acl)
Copied to clipboard
Hoang Phan, Xianjun Yang, Yuanshun Yao, Jingyu Zhang, Shengjie Bi, Xiaocheng Tang, Madian Khabsa, Lijuan Liu, Deren Lei
| Challenge: | Reinforcement learning with verifiable rewards (RLVR) has delivered impressive gains in mathematical and multimodal reasoning . however, the recipe introduces a significant risk of capability regression, where models forget foundational skills after prolonged training without employing regularization strategies. |
| Approach: | They propose a replay strategy with dynamic objective reweighting for general knowledge preservation using short-horizon signals of convergence and instability. |
| Outcome: | The proposed method preserves general capabilities and improves reasoning . it can be applied to existing RLVR pipelines without training additional models or tuning . |
Bridging Reasoning and Action: Hybrid LLM–RL Framework for Efficient Cross-Domain Task-Oriented Dialogue (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to solve cross-domain task-oriented dialogues are brittle when cross- domain constraints are not directly grounded in surface text or require commonsense inference. |
| Approach: | They propose a framework that makes LLM-derived constraint reasoning usable for RL. |
| Outcome: | Experiments show that the proposed framework outperforms single-model baselines on long-horizon tasks. |
Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing RLVR algorithms suffer from entropy collapse, leading to premature determinism and unstable optimization. |
| Approach: | They propose an adaptive entropy flow balancing mechanism that rescales entropic-increasing and enotro-decreazing updates according to their contributions to enthroy change. |
| Outcome: | The proposed method outperforms existing RLVR algorithms on six reasoning benchmarks. |