Graph-GRPO: Stabilizing Multi-Agent Topology Learning via Group Relative Policy Optimization (2026.findings-acl)
Copied to clipboard
Yueyang Cang, Xiaoteng Zhang, Erlu Zhao, Zehua Ji, Yuhang Liu, Yuchen He, Zhiyuan Ning, Chen Yijun, Wenge Que, Li Shi
| Challenge: | Recent approaches to optimize communication topology rely on single-sample policy gradients with absolute rewards. |
| Approach: | They propose a topology optimization framework that integrates Group Relative Policy Optimization. |
| Outcome: | The proposed topology optimization framework outperforms state-of-the-art methods on reasoning and code generation benchmarks. |
Similar Papers
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent preference-based fine-tuning methods have limited exploration in offline training . previous methods have been limited by the lack of exploration inherent in offline learning . |
| Approach: | They propose a method that normalizes rewards across a group of completed tasks to mitigate social bias in Large Language Models. |
| Outcome: | The proposed approach outperforms DPO and PPO in multiple benchmarks . it can overcome limitations of previous preference-based methods . |
DRA-GRPO: Your GRPO Needs to Know Diverse Reasoning Paths for Mathematical Reasoning (2026.findings-acl)
Copied to clipboard
Xiwen Chen, Wenhui Zhu, Peijie Qiu, Xuanzhao Dong, Hao Wang, Haiyu Wu, Huayu Li, Aris Sotiras, Yalin Wang, Abolfazl Razi
| Challenge: | Existing methods for group-relative policy optimization rely on scalar correctness rewards that are often non-injective with respect to semantic content. |
| Approach: | They propose a framework that calibrates the reward signal using the semantic density of sampled groups. |
| Outcome: | The proposed framework outperforms strong baselines on five math benchmarks with 7,000 samples and 55 cost. |
Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization (2026.acl-long)
Copied to clipboard
Yifeng Ding, Hung Le, Songyang Han, Kangrui Ruan, Zhenghui Jin, Varun Kumar, Zijian Wang, Anoop Deoras
| Challenge: | Current reinforcement learning methods suffer from coarse-grained, trajectory-level rewards that provide insufficient learning signals for complex multi-turn interactions, leading to training stagnation. |
| Approach: | They propose a novel RL algorithm for training large language models for multi-turn tool-integrated reasoning (TIR) that incorporates three innovations: turn-level reward assignment that provides fine-grained feedback for individual turns, return-based advantage estimation where normalized discounted returns are calculated as advantages, and self-supervised reward shaping that exploits self-supervision signals from generated code to densify sparse binary outcome-based rewards. |
| Outcome: | The proposed algorithm outperforms GRPO by 3.0% across diverse math reasoning benchmarks and improves grepo by 3.9% on commonsense reasoning and program synthesis tasks. |
Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to training large language models suffer from unstable value estimation, whereas outcome supervision struggles with credit assignment due to sparse, trajectory-level rewards. |
| Approach: | They propose a framework that integrates process supervision into group relative policy optimization. |
| Outcome: | The proposed framework outperforms standard GRPO on knowledge-intensive benchmarks by 5.0% and 6.3% on Qwen3-1.7B. |
MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can follow many natural-language instructions, yet they remain brittle when a request bundles multiple explicit constraints, such as asking the LLM to respond in a particular structure with an exact ending phrase. |
| Approach: | They propose a method which stabilizes learning through multi-temperature sampling to increase reward dispersion, dual-anchor advantages to restore gradients in homogeneous groups, prospect-theoretic shaping to bound updates and penalize violations based on Kahneman Tversky’s theory and asymmetric KL regularization. |
| Outcome: | The proposed method outperforms standard GRPO on FollowBench, IFEval, and a curated multi-constraint dataset, improving strict constraint satisfaction by up to 5.0% on Llama-3.2-3B. |
GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for group-relative policy optimization face challenges in reward sparsity, verbosity and inadequate focus on problem difficulty. |
| Approach: | They propose a method to improve group relative policy optimization with length-regularized rewards and explicit penalties for incorrect solutions. |
| Outcome: | The proposed method achieves state-of-the-art performance for 14B-scale models . it improves reasoning accuracy, conciseness, and efficiency . |
DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Data (2025.findings-emnlp)
Copied to clipboard
Yuhang Zhou, Jing Zhu, Shengyi Qian, Zhuokai Zhao, Xiyao Wang, Xiaoyu Liu, Ming Li, Paiheng Xu, Wei Ai, Furong Huang
| Challenge: | Large Language Models (LLMs) are increasingly aligned with human preferences through Reinforcement Learning from Human Feedback (RLHF). |
| Approach: | a new study proposes a domain-informed self-consistency policy optimization extension to GRPO that addresses inter-group imbalance. |
| Outcome: | a new extension of GRPO addresses inter-group imbalance with two key innovations . the proposed method outperforms existing GR PO variants by 5% on Qwen3 models . |
Auto-Weighted Group Relative Preference Optimization for Multi-Objective Text Generation Tasks (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Failing to balance the objectives in advance can lead to overfitting or insufficient learning of each reward function. |
| Approach: | They propose a method that adjusts reward weights according to learning progress . they evaluate AW-GRPO on advertising text generation problem . |
| Outcome: | The proposed method outperforms GRPO on advertising text generation tasks . it overfits BLEURT and jReadability at the expense of BLeurT performance . |
MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization (2026.acl-long)
Copied to clipboard
Yang Zhao, Hepeng Wang, Xiao Ding, Yangou Ouyang, Bibo Cai, Kai Xiong, Jinglong Gao, Zhouhao Sun, Li Du, Bing Qin, Ting Liu
| Challenge: | Group-Relative Policy Optimization (GRPO) has emerged as an efficient paradigm for aligning Large Language Models (LLMs), but its efficacy is confined to domains with verifiable ground truths. |
| Approach: | They propose a meta-cognitive orchestration layer that treats reward scalarization as a dynamic latent policy, leveraging the model’s terminal hidden states as 'a semantic bottleneck' . Across seven benchmarks, MAESTRO consistently outperforms single-reward and static multi-objective baselines while preserving the efficiency advantages of GRPO. |
| Outcome: | The proposed model outperforms single-reward and static multi-objective baselines while preserving efficiency advantages. |
MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning (2026.acl-long)
Copied to clipboard
Xiaoliang Fu, Jiaye Lin, Yangyi Fang, Binbin Zheng, Chaowen Hu, Zekai Shao, Cong Qin, Lu Pan, Ke Zeng, Xunliang Cai
| Challenge: | Existing RLVR algorithms rely on rigid, uniform, and symmetric trust region mechanisms . current algorithms lack robustness, asymmetric signal reliability and inefficient gradient utilization . |
| Approach: | They propose a framework to harmonize three dimensions of RLVR algorithms, a paper argues . a binary cutoff is used to discard valuable reinforcement signals, they argue . |
| Outcome: | The proposed framework outperforms baselines in evaluating a robust RLVR solution. |