Free Energy-Driven Reinforcement Learning with Adaptive Advantage Shaping for Unsupervised Reasoning in LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing unsupervised reinforcement learning methods lack the capacity to adapt to the model’s evolving reasoning capabilities during training. |
| Approach: | They propose an unsupervised reinforcement learning algorithm that adapts rewards to balance consensus and exploration based on the Free Energy Principle. |
| Outcome: | Empirical evaluations on nine datasets show that FREIA outperforms baseline methods on reasoning tasks. |
Similar Papers
Reinforcement Learning for Large Language Models via Group Preference Reward Shaping (2025.emnlp-main)
Copied to clipboard
Huaisheng Zhu, Siyuan Xu, Hangfan Zhang, Teng Xiao, Zhimeng Guo, Shijie Zhou, Shuyue Hu, Vasant G. Honavar
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) are expensive and sensitive to reward model quality. |
| Approach: | They propose a method that leverages preference-based comparisons rather than precise numerical rewards. |
| Outcome: | Experiments show that GPRS outperforms critic-model-free RL algorithms on RLHF and reasoning tasks. |
Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement Learning with Verifiable Rewards (RLVR) has shown significant promise for enhancing the reasoning capabilities of large language models (LLMs). |
| Approach: | They propose a model-free method that refines credit assignment by leveraging the model's internal uncertainty signals. |
| Outcome: | Extensive experiments on five mathematical reasoning benchmarks show that the proposed method outperforms strong RLVR baselines on multiple model scales, including 1.5B and 7B. |
Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work studies RLVR through token entropy, arguing that high-entropies drive exploration and should receive stronger updates. |
| Approach: | They propose a correctness-aware reinforcement framework that performs fine-grained advantage modulation over low-entropy segments. |
| Outcome: | The proposed framework improves accuracy over strong RL baselines across three backbones and six math benchmarks while maintaining high-entropy exploration. |
Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for reinforcement learning with verifiable rewards (RLVR) rely on static objective functions and rigid clipping strategies that misalign with the model’s evolving reasoning capabilities. |
| Approach: | They propose to incorporate Power-Mean Policy Optimization (PMPO) and Feedback-Adaptive Clipping (FAC) to overcome limitations of static mechanisms. |
| Outcome: | Extensive experiments on nine reasoning tasks show the proposed paradigm outperforms state-of-the-art methods. |
Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Reasoning ability is a defining capability of Large Language Models (LLMs), but RLVR training suffers from policy entropy collapse, hindering exploration and limiting reasoning performance. |
| Approach: | They propose a framework that dynamically balances exploration and exploitation via three components: difficulty-aware coefficient allocation, initial-anchored target entropy, and dynamic global coefficient adjustment. |
| Outcome: | The proposed framework outperforms baselines on multiple mathematical reasoning benchmarks. |
Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Instruction-fine-tuned large language models (LLMs) under 14B parameters underperform on NLU tasks . we explore a framework to improve the NLU capabilities of LLMs . |
| Approach: | They propose to use Proximal Policy Optimization to improve NLU capabilities . they frame NLU as a reinforcement learning environment and optimize for reward signals . |
| Outcome: | The proposed framework outperforms supervised fine-tuning on GLUE and superGLUE tasks. |
Breaking Consensus Bias: Unsupervised Reinforcement Learning for Machine Translation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing RL approaches for MT face fixed references or the production of homogeneous references leading to mode collapse in unsupervised settings. |
| Approach: | They propose an Entropy-Driven Unsupervised RL framework for machine translation that leverages entropy for supervision construction and self-evolution. |
| Outcome: | The proposed framework outperforms supervised and unsupervised baselines in multiple language pairs. |
Steering LLM Reasoning Through Bias-Only Adaptation (2025.emnlp-main)
Copied to clipboard
Viacheslav Sinii, Alexey Gorbatovski, Artem Cherepanov, Boris Shaposhnikov, Nikita Balagansky, Daniil Gavrilov
| Challenge: | Compared with LoRA and BitFit, training a single steering vector per layer with reinforcement learning requires orders of magnitude fewer resources and isolates a much smaller, more interpretable parameter set. |
| Approach: | They propose to train a single steering vector per layer with reinforcement learning while freezing all base weights to match the accuracy of fully RL-tuned reasoning models. |
| Outcome: | The proposed approach improves on an 8 billion-parameter model while keeping all base weights fixed. |
RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization (2026.acl-long)
Copied to clipboard
Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Yingwei MA, Jue Chen, Binhua Li, Zhi Jin, Fei Huang, Yongbin Li, Ge Li
| Challenge: | Reinforcement Learning with Verifiable Reward (RLVR) has significantly advanced the complex reasoning abilities of Large Language Models (LLMs). |
| Approach: | They propose a hybrid-policy optimization approach that synergizes internal exploitation with external data to achieve stronger reasoning capabilities. |
| Outcome: | The proposed approach achieves state-of-the-art performance on six math reasoning benchmarks and superior performance on out-of distribution reasoning tasks. |
Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches allocate an equal number of rollouts to all questions during the RL process, which is inefficient. |
| Approach: | They propose a mechanism for dynamically allocating rollout budgets based on the difficulty of the problems, enabling more efficient RL training. |
| Outcome: | The proposed model improves response precision while preserving exploratory ability to uncover potential correct pathways. |