Papers with PRM
Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models (2026.findings-acl)
Copied to clipboard
Teng Wang, Jiang Zhangyi, Zhenqi He, Hailei Gong, Shenyang Tong, Wenhan Yang, Zeyu Li, Yanan Zheng, Zifan He, Zewen Ye, Shengjie Ma, Jianping Zhang
| Challenge: | Existing Process Reward Models (PRMs) are vulnerable to reward hacking and require expensive, large-scale annotation of reasoning steps. |
| Approach: | They propose a reward model approach which evaluates both individual and consecutive reasoning steps from fine-grained and coarse-grounded level. |
| Outcome: | Empirical results show that the proposed model performs better than existing PRMs and is more robust than existing models. |
ProcessBench: Identifying Process Errors in Mathematical Reasoning (2025.acl-long)
Copied to clipboard
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin
| Challenge: | Existing models fail to generalize to more challenging math problems, authors say . existing benchmarks related to assessing language models' reasoning process are limited . |
| Approach: | They propose a tool to measure language models' ability to identify erroneous steps in reasoning . they use two types of models: process reward models and critic models . |
| Outcome: | The proposed model outperforms existing models in evaluating language models' reasoning process . the best open-source model has demonstrated the critique capability competitive with the proprietary model . |
Unearthing Gems from Stones: Policy Optimization with Negative Sample Augmentation for LLM Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in reasoning language models have witnessed a paradigm shift from short to long CoT pattern. |
| Approach: | They propose a behavior-constrained policy gradient with negative sample augmented (BCPG-NSA) negative steps are valuable components in long CoT models, authors argue . |
| Outcome: | The proposed framework outperforms baselines on math/coding reasoning benchmarks using the same training dataset. |
Learn from Relation Information: Towards Prototype Representation Rectification for Few-Shot Relation Extraction (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for Few-shot Relation Extraction focus on implicitly introducing relation information to constrain the prototype representation learning. |
| Approach: | They propose a parameter-less method to promote few-shot relation extraction . they use a prototype rectification module to rectify original prototypes by relation information . |
| Outcome: | The proposed method achieves state-of-the-art on fewRel 1.0 and 2.0 datasets. |
An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for constructing process supervision training data are costly or suffer from poor quality. |
| Approach: | They propose a framework called EpicPRM which annotates each intermediate reasoning step based on its quantified contribution and uses an adaptive binary search algorithm to enhance annotation precision and efficiency. |
| Outcome: | The proposed framework improves annotation precision and efficiency and can be used to train a high-quality training dataset with 50k annotated intermediate steps. |
Beyond the First Error: Process Reward Models for Reflective Mathematical Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for training effective PRMs focus on the first incorrect step and all preceding steps, assuming that all subsequent steps are incorrect. |
| Approach: | They propose a data annotation method specifically designed to score the long CoT reasoning process by using an LLM-based judger for annotation. |
| Outcome: | The proposed method improves PRMs' ability to identify effective self-correction behaviors and reasoning based on erroneous steps. |
Think Hard Only When Needed: A Hybrid Best-of-N and Beam Search for Efficient Test-Time Compute (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit remarkable reasoning and planning capabilities, yet their substantial inference-time cost significantly impedes deployment in resourceconstrained applications. |
| Approach: | They propose a hybrid inference pipeline that combines beam search and Best-of-N . THROW generates shorter initial trajectories and evaluates them using PRMs . |
| Outcome: | THROW achieves 1.54 and 14.38 latency speedups and 35.7% and 80.4% token reductions on average compared to Best-of-N and beam search . |
ViLBench: A Suite for Vision-Language Process Reward Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite its advantages, evaluation on PRMs remains less explored, especially in the multimodal domain. |
| Approach: | They propose to benchmark vision large language models as output reward models and process reward models as process-supervised reward models. |
| Outcome: | The proposed model outperforms both ORM and PRM on vision-language benchmarks and achieves an average improvement of 3.3% over standard CoT and up to 2.5% over its untrained counterpart on ViLBench. |
Efficient PRM Training Data Synthesis via Formal Verification (2026.findings-acl)
Copied to clipboard
Ryo Kamoi, Yusen Zhang, Nan Zhang, Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, Wenpeng Yin, Rui Zhang
| Challenge: | Existing approaches for constructing PRM training data rely on human annotation or sampling-based labeling methods that require repeated LLM calls. |
| Approach: | They propose a framework that synthesizes PRM training data by annotating step-level error labels using formal verification tools such as Z3 and Isabelle. |
| Outcome: | The proposed framework synthesizes PRM training data from formal logic and theorem proving tasks without human annotation or additional LLM calls. |
CodePRM: Execution Feedback-enhanced Process Reward Model for Code Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in code generation focus on optimizing the thought process, but lack effective process supervision, making it difficult to optimize the thoughts. |
| Approach: | They propose a method that leverages the code execution feedback to build a code PRM by collecting a large dataset of thought traces and then training it to take both the reasoning process and code execution as input. |
| Outcome: | The proposed approach outperforms baselines and strong LLMs in the inference stage. |
Retrieval-Augmented Process Reward Model for Generalizable Mathematical Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have advanced mathematical reasoning, but they still struggle with out-of-distribution (OOD) issues. |
| Approach: | They propose a framework to evaluate the logical validity of reasoning steps . they retrieves semantically similar questions and steps for PRM as a warmup . |
| Outcome: | The proposed framework outperforms baseline models on multiple real-world datasets. |
The Lessons of Developing Process Reward Models in Mathematical Reasoning (2025.findings-acl)
Copied to clipboard
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin
| Challenge: | a recent study shows that process reward models can make mistakes, leading to wrong conclusions. |
| Approach: | They propose a consensus filtering mechanism that integrates MC estimation with LLM-as-a-judge to improve model performance and data efficiency. |
| Outcome: | The proposed model outperforms existing open-source alternatives and provides practical guidelines for future research. |
The Bidirectional Process Reward Model (2026.acl-long)
Copied to clipboard
| Challenge: | Process reward models (PRMs) assign fine-grained scores to intermediate reasoning steps within a solution trajectory. |
| Approach: | They propose a bidirectional evaluation paradigm that integrates a parallel evaluation stream alongside the L2R evaluation scheme and a gating mechanism to fuse the reward scores. |
| Outcome: | The proposed model surpasses unidirectional baselines in multiple benchmarks, LLM objectives and sampling policies. |
ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents (2026.findings-acl)
Copied to clipboard
| Challenge: | Reward-guided search methods have shown potential in enhancing tool-using agents . however, there is a lack of reliable evaluation benchmarks for PRMs in tool-use settings . |
| Approach: | They propose a large-scale benchmark specifically designed to evaluate PRMs for tool-using agents. |
| Outcome: | The proposed benchmark shows that tool reward models perform better in tool-using environments. |
R-PRM: Reasoning-Driven Process Reward Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing Process Reward Models (PRMs) output evaluation scores directly, limiting both learning efficiency and evaluation accuracy. |
| Approach: | They propose a Reasoning-Driven Process Reward Modeling (R-PRM) which activates inherent reasoning to enhance process-level evaluation. |
| Outcome: | The proposed model outperforms baseline models on ProcessBench and PRMBench by 13.9 and 8.5 F1 scores. |
Better Process Supervision with Bi-directional Rewarding Signals (2025.findings-acl)
Copied to clipboard
Wenxiang Chen, Wei He, Zhiheng Xi, Honglin Guo, Boyang Hong, Jiazheng Zhang, Nijun Li, Tao Gui, Yun Li, Qi Zhang, Xuanjing Huang
| Challenge: | Existing processes that reward for each step are one-directional and lack a mechanism to model the distance to the final target. |
| Approach: | They propose a process supervision model that evaluates the correctness of previous steps and the probability of future success. |
| Outcome: | The proposed model outperforms existing supervision models like ORM and PRM on reasoning tasks and improves solution re-design. |
PRISM-MCTS: Learning from Reasoning Trajectories with Metacognitive Reflection (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing reasoning models are limited by inefficiency and computational redundancy . PRISM-MCTS integrates a process reward model with a dynamic shared memory . |
| Approach: | They propose a reasoning framework that integrates a process reward model with a dynamic shared memory. |
| Outcome: | PRISM-MCTS integrates a process reward model with a dynamic shared memory . it halves trajectory requirements on GPQA while surpassing MCTS-RAG and Search-o1 . |
Q-PRM: Adaptive Query Rewriting for Retrieval-Augmented Generation via Step-level Process Supervision (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to rewriting queries often lack supervision signals for intermediate steps . existing approaches rely on outcome-supervised training or heuristic rules to guide the rewrite process . |
| Approach: | They propose a query rewriting framework that generates process-level supervision signals for intermediate steps. |
| Outcome: | a new query rewriting framework outperforms existing approaches on open-domain QA benchmarks. |
Process-Supervised Reward Models for Verifying Clinical Note Generation: A Scalable Approach Guided by Domain Expertise (2025.emnlp-main)
Copied to clipboard
Hanyin Wang, Chufan Gao, Qiping Xu, Bolun Liu, Guleid Hussein, Hariprasad Reddy Korsapati, Mohamad El Labban, Kingsley Iheasirim, Mohamed Hassan, Gokhan Anil, Brian Bartlett, Jimeng Sun
| Challenge: | Currently, no automated, scalable method exists to evaluate the quality of LLM-generated clinical notes, leaving manual evaluation the gold standard. |
| Approach: | They propose a framework for training PRMs to deliver step-level reward signals for LLM-generated clinical notes. |
| Outcome: | The proposed framework outperforms reasoning and non-reasoning models on key evaluations and selects physician-preferred clinical notes with 56.2% accuracy. |
Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards (2026.acl-long)
Copied to clipboard
| Challenge: | Existing PRM datasets are expensive to construct and limited to the mathematical domain. |
| Approach: | They propose a method to generate a corpus of one million reasoning steps using the Planning Domain Definition Language. |
| Outcome: | The proposed model generates a corpus of approximately one million reasoning steps across various PDDL domains and trains them. |
Towards Inference-time Scaling for Continuous Space Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in reasoning large language models have expanded along training and inferencetime dimensions. |
| Approach: | They propose to use COCONUT (CITATION) continuous space reasoning LM as the backbone to generate diverse reasoning paths through dropout-based sampling. |
| Outcome: | The proposed method could enable performance gains similar to those observed in the discrete space, but only marginally improves in the continuous space. |
Search-in-Context: Efficient Multi-Hop QA over Long Contexts via Monte Carlo Tree Search with Dynamic KV Retrieval (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to multihop question answering (MHQA) over long contexts are often neglecting explicit reasoning or incurring expensive computational costs due to full-attention mechanisms over long contextuals. |
| Approach: | They propose a framework that integrates Monte Carlo Tree Search (MCTS) with dynamic key-value retrieval to enable iterative, context-aware reasoning. |
| Outcome: | The proposed framework integrates Monte Carlo Tree Search (MCTS) with dynamic key-value (KV) retrieval to enable iterative, context-aware reasoning. |
PRInTS: Reward Modeling for Long-Horizon Information Seeking (2026.acl-long)
Copied to clipboard
| Challenge: | Existing PRMs cannot capture richer dimensions of information-seeking steps, such as tool interactions and reasoning over tool outputs. |
| Approach: | They propose a generative PRM trained with dual capabilities that compresses the growing context while preserving essential information for step evaluation. |
| Outcome: | PRInTS improves on FRAMES, GAIA, and WebWalkerQA models while preserving essential information for step evaluation. |
Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have shown strong performance in many reasoning benchmarks, but lack robust planning or symbolic abstractions. |
| Approach: | They propose to synthesize high-quality symbolic reasoning trajectories with stepwise pseudo-labels at scale via Monte Carlo estimation. |
| Outcome: | The proposed method can be trained on high-quality symbolic reasoning trajectories with stepwise pseudo-labels at scale using Monte Carlo estimation. |
Efficient Process Reward Modeling via Contrastive Mutual Information (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to verify intermediate reasoning steps require human annotators to assign reward scores to each reasoning step, which is labor-intensive and costly. |
| Approach: | They propose a method that leverages the model's internal probability to infer step-level supervision while significantly reducing the computational burden of annotating dataset. |
| Outcome: | The proposed method reduces dataset construction time by 84% and token generation by 98% compared to MC estimation, while achieving higher accuracy on process-level evaluations and mathematical reasoning benchmarks. |
Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing reinforcement learning methods rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions. |
| Approach: | They propose a process reward model that rewards correct steps only when they detect errors . they propose VPPO, which rewards the correct prefix and an erroneous suffix . |
| Outcome: | a new approach outperforms sparse-reward RL and prior PRM-guided baselines on Pass@1 and Pass@K . a process reward model (PRM) outperformed sparser-rebound RL on multiple reasoning benchmarks . |
TRAC: Token-level Reward Assignment for Coherent Abstractive Summarization (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have achieved remarkable success in text summarization, but maintaining logical coherence and contextual consistency remains a pervasive challenge in long-form generation. |
| Approach: | They propose a framework that introduces a token-level reward function by integrating relative sentence gain, inter-sentence attention, and a Gaussian length penalty. |
| Outcome: | The proposed model outperforms the sequence-level baseline by 11.05% in fluency and 10.61% in Relevance. |
Valid Necessary: Diagnosing Latent Inefficiency in Chain-of-Thought (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing reasoning step evaluators fail to distinguish “valid but inefficient” reasoning steps from necessary reasoning. |
| Approach: | They propose a training-free metric that identifies low-utility steps and a post-hoc compression strategy to quantify their impact on token usage. |
| Outcome: | The proposed metric reduces token consumption by 31–53% while maintaining accuracy at substantially higher compression rates. |
Verified Critical Step Optimization for LLM Agents (2026.findings-acl)
Copied to clipboard
| Challenge: | Critical Step Optimization (CSO) focuses preference learning on verified critical steps where alternative actions demonstrably flip task outcomes from failure to success. |
| Approach: | They propose a method which focuses preference learning on verified critical steps where alternative actions demonstrably flip task outcomes from failure to success. |
| Outcome: | The proposed method outperforms the existing methods on GAIA-Text-103 and XBench-DeepSearch while requiring supervision at only 16% of trajectory steps. |
SAT: Balancing Reasoning Accuracy and Efficiency with Stepwise Adaptive Thinking (2026.acl-long)
Copied to clipboard
| Challenge: | Large Reasoning Models (LRMs) produce excessively long Chains of Thought (COT) Existing solutions that improve token efficiency but sacrifice fine-grained control can disrupt the logical integrity of the reasoning process. |
| Approach: | They propose a framework that performs step-level, difficulty-aware pruning while preserving the core reasoning structure. |
| Outcome: | Experiments show that SAT reduces reasoning tokens by 40% while maintaining or improving accuracy. |