Unearthing Gems from Stones: Policy Optimization with Negative Sample Augmentation for LLM Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in reasoning language models have witnessed a paradigm shift from short to long CoT pattern. |
| Approach: | They propose a behavior-constrained policy gradient with negative sample augmented (BCPG-NSA) negative steps are valuable components in long CoT models, authors argue . |
| Outcome: | The proposed framework outperforms baselines on math/coding reasoning benchmarks using the same training dataset. |
Similar Papers
Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing reinforcement learning methods rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions. |
| Approach: | They propose a process reward model that rewards correct steps only when they detect errors . they propose VPPO, which rewards the correct prefix and an erroneous suffix . |
| Outcome: | a new approach outperforms sparse-reward RL and prior PRM-guided baselines on Pass@1 and Pass@K . a process reward model (PRM) outperformed sparser-rebound RL on multiple reasoning benchmarks . |
Learning from Mistakes: Negative Reasoning Samples Enhance Out-of-Domain Generalization (2026.acl-long)
Copied to clipboard
Tian Xueyun, MingHua Ma, Bingbing Xu, Nuoyan Lyu, Wei Li, Heng Dong, Zheng Chu, Yuanzhuo Wang, Huawei Shen
| Challenge: | Recent studies show that supervised fine-tuning (SFT) is a common approach for reasoning in large language models. |
| Approach: | They propose to use supervised fine-tuning (SFT) on chain-of-thought trajectories demonstrations . they find that incorporating negative traxories yields substantial OOD generalization gains . |
| Outcome: | The proposed scheme yields 5.51% OOD gain over positive-only training. |
From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | Reinforcement learning with verifiable rewards (RLVR) has recently advanced the reasoning capabilities of large language models (LLMs). |
| Approach: | They propose a method that incorporates partial solution prefixes from expert demonstrations to guide the policy. |
| Outcome: | The proposed methods outperform strong baselines, yielding faster convergence and a higher performance ceiling. |
Fixing Distribution Shifts of LLM Self-Critique via On-Policy Self-Play Training (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models show impressive performance in a wide range of linguistic tasks, but their performance on complex reasoning tasks is still signif-icantly lower than the human level. |
| Approach: | They propose a reinforcement learning framework to synchronize the reasoning and critique capabilities of language models by using Monte Carlo sampling to give appropriate rewards to the model's critique content. |
| Outcome: | The proposed framework improves the model's reasoning and critique capabilities by 5.40 and 3.66 points, respectively, compared to the best baseline approach. |
Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) often produce unnecessarily long explanations that reduce efficiency. |
| Approach: | They propose a length-aware reward that selectively penalizes insignificance tokens . they also propose 'dynamic length control' that encourages more detailed reasoning . |
| Outcome: | The proposed method reduces response length while maintaining correctness, the authors show . it selectively penalizes insignificance tokens while maintaining accuracy . |
Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion (2024.emnlp-main)
Copied to clipboard
Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Eugene Choi, Bill Wu, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Azar, Olivier Pietquin, Matthieu Geist
| Challenge: | Reinforcement Learning (RL) is a method used to fine tune Large Language Models (LLMs) using a reward model trained from preference data to better align with human judgment. |
| Approach: | They propose a Reinforcement Learning (RL) algorithm that can estimate the optimal policy even from off-policy data. |
| Outcome: | The proposed algorithm can estimate the optimal policy even from off-policy data. |
Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to RLVR provide sparse supervision since reward arrives only after the full generation is complete. |
| Approach: | They propose a step-level reward system that extracts confidence and correctness and combines them into a Step Potential signal that explicitly estimates reasoning state at each step. |
| Outcome: | The proposed method outperforms existing methods on multiple benchmarks and improves accuracy while reducing response length. |
Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards (2026.acl-long)
Copied to clipboard
| Challenge: | Large reasoning models are typically trained using reinforcement learning with verifiable reward (RLVR) positive and negative self-generated rollouts are used to update the model's policy . positive samples sharpen existing correct reasoning patterns, while negative samples encourage exploration of new reasoning paths. |
| Approach: | They propose a method that allocates advantage signals to key tokens across different polarities. |
| Outcome: | The proposed method improves the ability of large reasoning models to learn from their own generated rollouts. |
Correct, Concise and Complete: Multi-stage Training For Adaptive Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) increase test-time computation, often in the form of chain-of-thought (CoT) however, reasoning traces can become unnecessarily long, increasing computation costs without improving accuracy and sometimes even degrading performance. |
| Approach: | They propose a multi-stage efficient reasoning method that combines supervised fine-tuning with reinforcement learning using an adaptive length penalty. |
| Outcome: | The proposed method reduces response length by an average of 28% for 8B models and 40% for 32B models while incurring only minor performance drops of 1.6 and 2.5 points, respectively. |
Ada-RS: Adaptive Rejection Sampling for Selective Thinking (2026.acl-industry)
Copied to clipboard
Yirou Ge, Yixi Li, Alec M. Chiu, Shivani Shekhar, Zijie Pan, Avinash Thangali, Yun-Shiuan Chuang, Chaitanya Kulkarni, Uma Kona, Linsey Pang, Prakhar Mehrotra
| Challenge: | Large language models are increasingly being deployed in cost- and latency-sensitive settings . chain-of-thought improves reasoning, but it can waste tokens on simple requests . |
| Approach: | They introduce an algorithm-agnostic sample filtering framework for learning selective reasoning . they show that Ada-RS reduces average output tokens by 80% and reducing thinking rate by 5% . |
| Outcome: | The proposed framework reduces output tokens by 80% and thinking rate by 95% on a synthetic tool call-oriented e-commerce benchmark. |