| Challenge: | Recent work has aimed to enhance reasoning capabilities of language models, but these methods are limited to domains with objectively verifiable answers. |
| Approach: | They propose a self-play framework to improve reasoning on general-domain data. |
| Outcome: | Experiments show that the proposed framework improves reasoning performance on general-domain data while maintaining competitive performance on verifiable academic benchmarks. |
Similar Papers
Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to improve long-chain mathematical reasoning focus on the first erroneous step, but ignore all other steps and rely heavily on external signals. |
| Approach: | They propose a DPO framework that leverages step-wise rewards from the entire reasoning chain instead of optimizing only the first erroneous step. |
| Outcome: | The proposed framework improves on in-domain and out-of-domain mathematical reasoning benchmarks. |
DRA-GRPO: Your GRPO Needs to Know Diverse Reasoning Paths for Mathematical Reasoning (2026.findings-acl)
Copied to clipboard
Xiwen Chen, Wenhui Zhu, Peijie Qiu, Xuanzhao Dong, Hao Wang, Haiyu Wu, Huayu Li, Aris Sotiras, Yalin Wang, Abolfazl Razi
| Challenge: | Existing methods for group-relative policy optimization rely on scalar correctness rewards that are often non-injective with respect to semantic content. |
| Approach: | They propose a framework that calibrates the reward signal using the semantic density of sampled groups. |
| Outcome: | The proposed framework outperforms strong baselines on five math benchmarks with 7,000 samples and 55 cost. |
GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent reinforcement learning approaches have advanced reasoning in Large Language Models (LLMs), yet their adaptation to multimodal LLMs remains underexplored. |
| Approach: | They propose a reinforcement learning framework that eliminates KL penalties and rewards consistency . they propose GRPO-CARE, which outperforms standard GR PO, with a base reward for accuracy and an adaptive bonus for consistency. |
| Outcome: | The proposed framework outperforms standard GRPO on the most difficult evaluation level and reasoning consistency test benchmarks. |
CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown great potential in natural language processing tasks, but their application to machine translation remains challenging due to pretraining on predominantly English-centric datasets. |
| Approach: | They propose a method that combines reward scores with model confidence to improve model selection for fine-tuning. |
| Outcome: | The proposed method outperforms existing methods in translation accuracy and data efficiency. |
TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization (2026.findings-acl)
Copied to clipboard
Shichao Ma, Zhiyuan Ma, Ming Yang, Xiaofan Li, Xing Wu, Jintao Du, Yu Cheng, Weiqiang Wang, Qiliang Liu, Zhengyang Zhou, Yang Wang
| Challenge: | Large Language Models (LLMs) can solve complex tasks through iterative information retrieval. |
| Approach: | They propose a turn-level stage-aware policy optimization approach to solve this problem . they introduce a first-occurrence latent reward mechanism to allocate partial rewards . |
| Outcome: | Experiments show that TSPO outperforms state-of-the-art models on Qwen2.5-3B and 7B models. |
Step-level Value Preference Optimization for Mathematical Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating preference-level annotations do not capture the fine-grained quality of model outputs in multi-step reasoning tasks. |
| Approach: | They propose an algorithm to automatically annotate step-level preferences for multi-step reasoning using Monte Carlo Tree Search. |
| Outcome: | The proposed algorithm achieves state-of-the-art performance on in-domain and out-of domain mathematical reasoning benchmarks. |
Reasoning Paths Optimization: Learning to Reason and Explore From Diverse Paths (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Advanced models such as OpenAI o1 exhibit impressive problem-solving capabilities, but they may still falter on more complex problems, making errors that disrupt their reasoning paths. |
| Approach: | They propose a framework that encourages favorable branches at each reasoning step while penalizing unfavorable ones, enhancing the model’s overall problem-solving performance. |
| Outcome: | The proposed framework improves reasoning performance on multi-step reasoning tasks such as math word problems and science-based exam questions. |
DPEPO: Diverse Parallel Exploration Policy Optimization for LLM-based Agents (2026.acl-long)
Copied to clipboard
JunShuo Zhang, Chengrui Huang, Feng Guo, Zihan Li, Ke Shi, Menghua Jiang, Jiguo Yu, Shuo Shang, Shen Gao
| Challenge: | Existing approaches to large language model (LLM) agents that follow the sequential "reason-then-act" paradigm suffer from limited exploration and incomplete environmental understanding as they interact with only a single environment per step. |
| Approach: | They propose a paradigm that enables an agent to interact with multiple environments simultaneously and share cross-trajectory experiences. |
| Outcome: | The proposed paradigm achieves state-of-the-art (SOTA) success rates while maintaining comparable efficiency to strong sequential baselines. |
Playpen: An Environment for Exploring Learning From Dialogue Game Feedback (2025.emnlp-main)
Copied to clipboard
Nicola Horst, Davide Mazzaccara, Antonia Schmidt, Michael Sullivan, Filippo Momentè, Luca Franceschetti, Philipp Sadler, Sherzod Hakimov, Alberto Testoni, Raffaella Bernardi, Raquel Fernández, Alexander Koller, Oliver Lemon, David Schlangen, Mario Giulianelli, Alessandro Suglia
| Challenge: | In this paper, we investigate whether Dialogue Games—goal-directed and rule-governed activities driven predominantly by verbal actions—can also serve as a source of feedback signals for learning. |
| Approach: | They introduce Playpen, an environment for off- and online learning through Dialogue Game self-play, and investigate a representative set of post-training methods: supervised fine-tuning, direct alignment and reinforcement learning with Group Relative Policy Optimization. |
| Outcome: | The proposed model improves performance on unseen instances, but negatively impacts other skills, while interactive learning shows balanced improvements without loss of skills. |
RADO: Reasoning Audit-Driven Optimization for Rigorous Reasoning in High-Stakes Domains (2026.acl-long)
Copied to clipboard
| Challenge: | Current reinforcement learning paradigms rely on outcome-based rewards, overlooking latent logical fallacies in intermediate steps. |
| Approach: | They propose a specialized audit model augmented with external tools to identify local logical ruptures and calibrate reward signals. |
| Outcome: | The proposed framework improves accuracy and logical rigor in high-stakes domains. |