Papers with RLIF
A Few Bad Apples Spoil the Bunch: Preventing Global Entropy Collapse Driven by a Small Set of Tokens in LLM Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement learning with verifiable rewards and Reinforced Learning from internal feedback fail to benefit from test-time compute due to entropy collapse and the resulting loss of reasoning diversity. |
| Approach: | They propose a strategy that assigns each generated token a redistribution score and applies selective KL regularization to only the top 5% of tokens under this score. |
| Outcome: | The proposed model improves on both RLVR and RLIF models on math reasoning benchmarks, showing that targeted entropy control at a vanishingly small subset of tokens is sufficient to sustain reasoning diversity and effective test-time scaling. |
Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning tasks and general tasks. |
| Approach: | They propose a "Verifier-free Intrinsic Gradient-Norm Reward" that uses only the policy model itself. |
| Outcome: | The proposed reward outperforms the state-of-the-art RLIF baseline INTUITOR on math benchmarks and shows cross-domain transfer to code benchmarks when trained only on math data. |