MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have demonstrated remarkable capabilities in mathematical reasoning tasks. |
| Approach: | They propose to use Maximal Marginal Relevance to reweigh rewards of multiple rollouts by balancing rollout quality with diversity to reduce rollout redundancy. |
| Outcome: | The proposed approach reduces training time and costs by 47.9% . evaluations across three model sizes, three GRPO variants, and five mathematical reasoning benchmarks show that it achieves comparable peak performance while requiring on average 70.2% less wall-clock time. |
Similar Papers
DRA-GRPO: Your GRPO Needs to Know Diverse Reasoning Paths for Mathematical Reasoning (2026.findings-acl)
Copied to clipboard
Xiwen Chen, Wenhui Zhu, Peijie Qiu, Xuanzhao Dong, Hao Wang, Haiyu Wu, Huayu Li, Aris Sotiras, Yalin Wang, Abolfazl Razi
| Challenge: | Existing methods for group-relative policy optimization rely on scalar correctness rewards that are often non-injective with respect to semantic content. |
| Approach: | They propose a framework that calibrates the reward signal using the semantic density of sampled groups. |
| Outcome: | The proposed framework outperforms strong baselines on five math benchmarks with 7,000 samples and 55 cost. |
GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for group-relative policy optimization face challenges in reward sparsity, verbosity and inadequate focus on problem difficulty. |
| Approach: | They propose a method to improve group relative policy optimization with length-regularized rewards and explicit penalties for incorrect solutions. |
| Outcome: | The proposed method achieves state-of-the-art performance for 14B-scale models . it improves reasoning accuracy, conciseness, and efficiency . |
Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization (2026.acl-long)
Copied to clipboard
Yifeng Ding, Hung Le, Songyang Han, Kangrui Ruan, Zhenghui Jin, Varun Kumar, Zijian Wang, Anoop Deoras
| Challenge: | Current reinforcement learning methods suffer from coarse-grained, trajectory-level rewards that provide insufficient learning signals for complex multi-turn interactions, leading to training stagnation. |
| Approach: | They propose a novel RL algorithm for training large language models for multi-turn tool-integrated reasoning (TIR) that incorporates three innovations: turn-level reward assignment that provides fine-grained feedback for individual turns, return-based advantage estimation where normalized discounted returns are calculated as advantages, and self-supervised reward shaping that exploits self-supervision signals from generated code to densify sparse binary outcome-based rewards. |
| Outcome: | The proposed algorithm outperforms GRPO by 3.0% across diverse math reasoning benchmarks and improves grepo by 3.9% on commonsense reasoning and program synthesis tasks. |
Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening (2025.emnlp-main)
Copied to clipboard
| Challenge: | Reinforcement learning is emerging as a primary driver for improving language model reasoning capabilities. |
| Approach: | They propose a method for explicitly up-weighting rare but correct solutions to overcome rank bias in group relative policy optimization (GRPO) . |
| Outcome: | The proposed method mitigates rank bias and improves pass@N across a large range of N in both synthetic and real theorem proving settings. |
N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that Large Language Models can generate diverse solutions during the rollout phase. |
| Approach: | They propose a new approach that leverages Semantic Neighbor Mixing to generate diverse input representations by mixing anchor tokens and nearest semantic neighbors. |
| Outcome: | Experimental results show that the proposed approach improves on strong baselines and generalizes on out-of-distribution tasks. |
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent preference-based fine-tuning methods have limited exploration in offline training . previous methods have been limited by the lack of exploration inherent in offline learning . |
| Approach: | They propose a method that normalizes rewards across a group of completed tasks to mitigate social bias in Large Language Models. |
| Outcome: | The proposed approach outperforms DPO and PPO in multiple benchmarks . it can overcome limitations of previous preference-based methods . |
GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent reinforcement learning approaches have advanced reasoning in Large Language Models (LLMs), yet their adaptation to multimodal LLMs remains underexplored. |
| Approach: | They propose a reinforcement learning framework that eliminates KL penalties and rewards consistency . they propose GRPO-CARE, which outperforms standard GR PO, with a base reward for accuracy and an adaptive bonus for consistency. |
| Outcome: | The proposed framework outperforms standard GRPO on the most difficult evaluation level and reasoning consistency test benchmarks. |
On the Hidden Objective Biases of Group-based Reinforcement Learning (2026.acl-short)
Copied to clipboard
| Challenge: | Recent studies have reported unexpected behaviors during training, including lengthrelated biases, formatting tokens, and reward hacking in multi-objective settings. |
| Approach: | They propose to analyze group-based reinforcement learning methods within a unified surrogate formulation. |
| Outcome: | The proposed methods exhibit structural mismatches between reward optimization and the underlying training objective. |
Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning (2026.acl-long)
Copied to clipboard
Ziheng Li, Liu Kang, Feng Xiao, Luxi Xing, Qingyi Si, Zhuoran Li, Weikang Gong, Deqing Yang, Yanghua Xiao, Hongcheng Guo
| Challenge: | Group Relative Policy Optimization (GRPO) uses a coarse-grained credit assignment mechanism that propagates group-level rewards uniformly to to every token in a sequence, neglecting the varying contribution of individual reasoning steps. |
| Approach: | They introduce Outcome-grounded Advantage Reshaping (OAR) which redistributes advantages based on how much each token influences the model’s final answer. |
| Outcome: | Empirical results show that OAR-G outperforms GRPO on a high-fidelity attribution signal and suppresses low-impact tokens while preserving the advantage mass. |
Auto-Weighted Group Relative Preference Optimization for Multi-Objective Text Generation Tasks (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Failing to balance the objectives in advance can lead to overfitting or insufficient learning of each reward function. |
| Approach: | They propose a method that adjusts reward weights according to learning progress . they evaluate AW-GRPO on advertising text generation problem . |
| Outcome: | The proposed method outperforms GRPO on advertising text generation tasks . it overfits BLEURT and jReadability at the expense of BLeurT performance . |