From log 𝜋 to 𝜋: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight (2026.acl-long)
Copied to clipboard
Xiaoliang Fu, Jiaye Lin, Yangyi Fang, Chaowen Hu, Cong Qin, Zekai Shao, Binbin Zheng, Lu Pan, Ke Zeng
| Challenge: | Standard algorithms for Large Language Models (LLMs) enforce stability via "hard clipping" but relying on log-probability gradient yields divergent weights as probabilities vanish, destabilizing LLM training. |
| Approach: | They propose a decoupled gradient policy optimization that uses a decay mechanism to decouple the probability of a boundary token. |
| Outcome: | The proposed algorithm outperforms baselines on various mathematical benchmarks. |
Similar Papers
MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning (2026.acl-long)
Copied to clipboard
Xiaoliang Fu, Jiaye Lin, Yangyi Fang, Binbin Zheng, Chaowen Hu, Zekai Shao, Cong Qin, Lu Pan, Ke Zeng, Xunliang Cai
| Challenge: | Existing RLVR algorithms rely on rigid, uniform, and symmetric trust region mechanisms . current algorithms lack robustness, asymmetric signal reliability and inefficient gradient utilization . |
| Approach: | They propose a framework to harmonize three dimensions of RLVR algorithms, a paper argues . a binary cutoff is used to discard valuable reinforcement signals, they argue . |
| Outcome: | The proposed framework outperforms baselines in evaluating a robust RLVR solution. |
How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization (2026.findings-acl)
Copied to clipboard
Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi, Chaowen Hu, Lu Pan, Ke Zeng, Xunliang Cai
| Challenge: | Existing methods for reinforcement learning with verifiable rewards are limited by the complexity of the problem and the complexity. |
| Approach: | They propose a theoretically-grounded dual-pronged optimization framework for reinforcement learning with verifiable rewards that compensates for gradient attenuation of high-confidence correct actions while utilizing entropy changes as computable indicators to stabilize excessive update magnitudes. |
| Outcome: | The proposed framework compensates for gradient attenuation of high-confidence correct actions while utilizing entropy changes as computable indicators to stabilize excessive update magnitudes. |
RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization (2026.acl-long)
Copied to clipboard
Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Yingwei MA, Jue Chen, Binhua Li, Zhi Jin, Fei Huang, Yongbin Li, Ge Li
| Challenge: | Reinforcement Learning with Verifiable Reward (RLVR) has significantly advanced the complex reasoning abilities of Large Language Models (LLMs). |
| Approach: | They propose a hybrid-policy optimization approach that synergizes internal exploitation with external data to achieve stronger reasoning capabilities. |
| Outcome: | The proposed approach achieves state-of-the-art performance on six math reasoning benchmarks and superior performance on out-of distribution reasoning tasks. |
CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for proximal policy optimization discard valuable gradient signals from low-probability tokens due to the clipping mechanism. |
| Approach: | They propose an algorithm that reintroduces gradients from clipped tokens in native PPO in a gentle and bounded manner. |
| Outcome: | The proposed algorithm outperforms strong baselines on reasoning benchmarks on different model scales. |
Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward (2026.findings-acl)
Copied to clipboard
Guanhua Huang, Tingqiang Xu, Mingze Wang, Qi Yi, Xue Gong, Siheng Li, Ruibin Xiong, Kejiao Li, Yuhao Jiang, Bo Zhou
| Challenge: | Recent studies show that RLVR training is slow and results plateau as policy entropy collapses . low-probability regularization (Lp-Reg) reduces the number of low-quality exploratory tokens induced by RL training . |
| Approach: | They propose a method to reduce RLVR over-penalization by eliminating low-probability exploratory tokens . they propose 'Low-provability Regularization' to reduce the gradual elimination of low-quality exploratory entropy tokens. |
| Outcome: | The proposed method eliminates low-probability exploratory tokens and prevents suppression of potentially valuable low-property candidates. |
Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing RLVR algorithms suffer from entropy collapse, leading to premature determinism and unstable optimization. |
| Approach: | They propose an adaptive entropy flow balancing mechanism that rescales entropic-increasing and enotro-decreazing updates according to their contributions to enthroy change. |
| Outcome: | The proposed method outperforms existing RLVR algorithms on six reasoning benchmarks. |
Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement Learning with Verifiable Rewards (RLVR) has shown significant promise for enhancing the reasoning capabilities of large language models (LLMs). |
| Approach: | They propose a model-free method that refines credit assignment by leveraging the model's internal uncertainty signals. |
| Outcome: | Extensive experiments on five mathematical reasoning benchmarks show that the proposed method outperforms strong RLVR baselines on multiple model scales, including 1.5B and 7B. |
Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for reinforcement learning with verifiable rewards (RLVR) rely on static objective functions and rigid clipping strategies that misalign with the model’s evolving reasoning capabilities. |
| Approach: | They propose to incorporate Power-Mean Policy Optimization (PMPO) and Feedback-Adaptive Clipping (FAC) to overcome limitations of static mechanisms. |
| Outcome: | Extensive experiments on nine reasoning tasks show the proposed paradigm outperforms state-of-the-art methods. |
LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment (2026.findings-acl)
Copied to clipboard
Shipeng Li, Zhiqin Yang, Shikun Li, Xiaobo Xia, Hengyu Liu, Xinghua Zhang, Gaode Chen, Dong Fang, Ying Tai, Zhe Peng
| Challenge: | Reinforcement learning with verifiable rewards (RLVR) is a key technique for enhancing LLMs’ reasoning abilities, yet its data inefficiency remains a major bottleneck. |
| Approach: | They propose a gradient-alignment-based method which intelligently selects the learnable and representative training reasoning data for RLVR post-training. |
| Outcome: | Experiments on five reasoning benchmarks show that the proposed method significantly reduces training data requirements while improving performance. |
Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing unified optimization strategies overlook the statistical conflict between these distinct gradient signals. |
| Approach: | They propose a framework to reduce bias-variance trade-offs in Large Language Models . they propose DYPO, which leverages intrinsic group dynamics to significantly reduce RL gradient variance . |
| Outcome: | The proposed framework outperforms traditional pipelines on reasoning benchmarks and out-of-distribution tasks. |