Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts (2026.acl-long)
Copied to clipboard
| Challenge: | Reinforcement learning with verifiable rewards (RLVR) training with Mixture-of-Experts policies remains fragile and prone to reward collapse. |
| Approach: | They propose a router shift-based policy optimization method that computes a per-token router-shift ratio conditioned on the previously activated experts and applies stop-gradient and a lower-bound floor. |
| Outcome: | The proposed method achieves better performance and greater stability than previous methods. |
Similar Papers
Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing RLVR algorithms suffer from entropy collapse, leading to premature determinism and unstable optimization. |
| Approach: | They propose an adaptive entropy flow balancing mechanism that rescales entropic-increasing and enotro-decreazing updates according to their contributions to enthroy change. |
| Outcome: | The proposed method outperforms existing RLVR algorithms on six reasoning benchmarks. |
Improved Policy Optimization for Mixture-of-Experts Models: Importance Sampling and Rewarding from an Expert-Centric Perspective (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to reinforcement learning (RL) suffer from training instability . existing approaches often ignore token-specific discrepancies in expert assignments . |
| Approach: | They propose to introduce expert-level importance sampling to reduce complexity of RL . they propose to leverage expert-centric granularity to ensure a rigorous alignment between reward signals and policy updates. |
| Outcome: | The proposed method outperforms strong baselines across reasoning tasks. |
Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing RLVR algorithms focus on different granularities and have complementary strengths and limitations. |
| Approach: | They propose a framework for reinforcement learning with verifiable rewards that bridges RLVR and GSPO . group-level importance ratios are used to update a policy, which preserves fine-grained credit assignment . |
| Outcome: | The proposed framework outperforms existing methods on seven reasoning benchmarks. |
Think Outside the Policy: In-Context Steered Policy Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing Reinforcement Learning from Verifiable Rewards (RLVR) methods exhibit limited exploration due to reliance on on-policy rollouts which are limited to the current policy’s distribution, resulting in narrow trajectory diversity. |
| Approach: | They propose a framework that leverages the in-context learning capability of Large Reasoning Models to provide expert guidance using existing datasets. |
| Outcome: | The proposed framework improves RLVR performance and training stability on mathematical reasoning benchmarks. |
Exploration-Driven Reinforcement Learning for Expert Routing Improvement in Mixture-of-Experts Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | MoE-based LLMs are not explicitly supervised to select suitable experts. |
| Approach: | They propose Exploration-Driven Reinforcement Learning (ERL) which explicitly optimizes the router by exploration of alternative routing paths. |
| Outcome: | The proposed method improves summarization (SAMSum, XSUM, question answering, and language modeling), and raises routing quality, delivering 8.9 higher MRR than baselines over 100 perturbed routing paths. |
From Pseudo-Balancing to True Specialization: Memory-Aware Routing for Mixture-of-Experts (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to optimize expert-centered load balancing fail to account for pseudo-balance phenomenon . severe knowledge overlap among experts leads to redundant representations and inefficient parameter utilization . |
| Approach: | They propose a method that prioritizes expert utilization over semantic alignment . they use memory-aware routing to ensure expert load balancing is consistent . |
| Outcome: | Experimental results show that MAR improves expert specialization by 35% and accuracy by 2%-25% . MAR matches baseline performance with only half the experts . |
How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization (2026.findings-acl)
Copied to clipboard
Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi, Chaowen Hu, Lu Pan, Ke Zeng, Xunliang Cai
| Challenge: | Existing methods for reinforcement learning with verifiable rewards are limited by the complexity of the problem and the complexity. |
| Approach: | They propose a theoretically-grounded dual-pronged optimization framework for reinforcement learning with verifiable rewards that compensates for gradient attenuation of high-confidence correct actions while utilizing entropy changes as computable indicators to stabilize excessive update magnitudes. |
| Outcome: | The proposed framework compensates for gradient attenuation of high-confidence correct actions while utilizing entropy changes as computable indicators to stabilize excessive update magnitudes. |
From log 𝜋 to 𝜋: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight (2026.acl-long)
Copied to clipboard
Xiaoliang Fu, Jiaye Lin, Yangyi Fang, Chaowen Hu, Cong Qin, Zekai Shao, Binbin Zheng, Lu Pan, Ke Zeng
| Challenge: | Standard algorithms for Large Language Models (LLMs) enforce stability via "hard clipping" but relying on log-probability gradient yields divergent weights as probabilities vanish, destabilizing LLM training. |
| Approach: | They propose a decoupled gradient policy optimization that uses a decay mechanism to decouple the probability of a boundary token. |
| Outcome: | The proposed algorithm outperforms baselines on various mathematical benchmarks. |
MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning (2026.acl-long)
Copied to clipboard
Xiaoliang Fu, Jiaye Lin, Yangyi Fang, Binbin Zheng, Chaowen Hu, Zekai Shao, Cong Qin, Lu Pan, Ke Zeng, Xunliang Cai
| Challenge: | Existing RLVR algorithms rely on rigid, uniform, and symmetric trust region mechanisms . current algorithms lack robustness, asymmetric signal reliability and inefficient gradient utilization . |
| Approach: | They propose a framework to harmonize three dimensions of RLVR algorithms, a paper argues . a binary cutoff is used to discard valuable reinforcement signals, they argue . |
| Outcome: | The proposed framework outperforms baselines in evaluating a robust RLVR solution. |
Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective (2026.acl-long)
Copied to clipboard
Zhezheng Hao, Hong Wang, Haoyang Liu, Jian Luo, Jiarui Yu, Hande Dong, Qiang Lin, Can Wang, Jiawei Chen
| Challenge: | Large Language Models (LLMs) have remarkable reasoning capabilities in complex tasks such as mathematics and coding. |
| Approach: | They propose an entropy-modulation method that adaptively reweighs tokens based on theoretically-estimated entropic variations. |
| Outcome: | The proposed method outperforms state-of-the-art methods in six mathematical reasoning and three coding benchmarks. |