ToolRM: Towards Agentic Tool-Use Reward Modeling

Challenge:	lack of reliable reward models for tool-use tasks has limited progress toward agentic AI . recent advances in agentic artificial intelligence are driven by tool-using capabilities of large language models.
Approach:	They propose a pipeline that constructs pairwise preference data using rule-based scoring and multidimensional sampling to build lightweight reward models.
Outcome:	The proposed model outperforms existing models on tool calling tasks with higher accuracy.

Similar Papers

Overview

Challenge

Approach

Outcome

Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling (2026.acl-long)

Copied to clipboard

Jiaxuan Wang, Yulan Hu, Wenjin Yang, Zheng Pan, Xin Li, Lan-Zhe Guo

Challenge:	Large Language Models (LLMs) evolve into agentic systems capable of autonomous tool invocation and complex reasoning.
Approach:	They propose a trajectory-level preference benchmark to evaluate judges' ability to distinguish preferred versus distractor agent trajectories in tool-integrated environments.
Outcome:	The proposed benchmark evaluates how well judges distinguish preferred versus distractor agent trajectories in complex tool-using scenarios.

RewardBench: Evaluating Reward Models for Language Modeling (2025.findings-naacl)

Copied to clipboard

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi

Language Models

Challenge:	Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models.
Approach:	They present a benchmark dataset and code-base for evaluation of reward models . they use prompt-chosen-rejected trios to benchmark how they perform on queries .
Outcome:	The proposed dataset compares RMs with other models on a set of questions.

PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling (2026.acl-long)

Copied to clipboard

Ai Jian, Jingqing Ruan, Xing Ma, Dailin Li, Weipeng Zhang, Ke Zeng, Xunliang Cai

Challenge:	Existing reward models lack generative and reasoning capabilities, resulting in poor performance.
Approach:	They propose a reward-aware task-adaptive reward model that enables pointwise training using readily available pairwise data via a novel Preference-Aware Reward mechanism.
Outcome:	The proposed reward model achieves an average relative improvement of 8.7% over the base models on RewardBench and RMBench.

ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents (2026.findings-acl)

Copied to clipboard

Dawei Li, Yuguang Yao, Zhen Tan, Huan Liu, Ruocheng Guo

Challenge:	Reward-guided search methods have shown potential in enhancing tool-using agents . however, there is a lack of reliable evaluation benchmarks for PRMs in tool-use settings .
Approach:	They propose a large-scale benchmark specifically designed to evaluate PRMs for tool-using agents.
Outcome:	The proposed benchmark shows that tool reward models perform better in tool-using environments.

ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework (2026.acl-long)

Copied to clipboard

Kai Qin, Liangxin Liu, Yu Liang, Longzheng Wang, null Wangyan, Zhang Yueyang, Long Xia, Zhiyuan Sun, Houde Liu, Daiting Shi

Challenge:	Existing methods for generating reward models focus on outcome-level supervision, neglecting analytical process quality, which constrains their potential.
Approach:	They propose a novel reward model that leverages self-reflection to assess analytical quality and enhance preference modeling.
Outcome:	The proposed model improves performance on four benchmarks and significantly mitigates positional bias.

Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems (2025.acl-long)

Copied to clipboard

Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu, Lei Hou, Juanzi Li

Challenge:	Existing reward models focus on human preferences, neglecting verifiable correctness signals.
Approach:	They propose a reward system that combines human preference rewards with verifiable correctness signals to provide reliable rewards.
Outcome:	The proposed reward agent significantly outperforms vanilla reward models on benchmarks and inference-time best-of-n searches on real-world tasks.

Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents (2025.acl-long)

Copied to clipboard

Tianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

Multimodality

Challenge:	Multimodal Large Language Models (MLLMs) are developing but lack external feedback . there is no clear on how to select reward models for agents .
Approach:	They propose a benchmark to evaluate agent reward modeling ability in MLLMs . they use multiple dimensions and real-world agent scenarios evaluation .
Outcome:	The proposed benchmark evaluates agent performance in multimodal large language models . it covers perception, planning, and safety with 7 scenarios and is highly difficult and high-quality .

M-RewardBench: Evaluating Reward Models in Multilingual Settings (2025.acl-long)

Copied to clipboard

Srishti Gureja, Lester James Validad Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Triandi Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, Marzieh Fadaee

Multilinguality

Challenge:	Reward models (RMs) are primarily trained and evaluated in English and their capabilities in multilingual settings remain understudied.
Approach:	They construct a multilingual RM evaluation benchmark that tests the chat, safety, reasoning, and translation capabilities of RMs in 23 languages.
Outcome:	The proposed model performs better for high-resource languages and improves with translation quality.

APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport (2025.emnlp-main)

Copied to clipboard

Zhuo Li, Yuege Feng, Dandan Guo, Jinpeng Hu, Anningzhe Gao, Xiang Wan

Robustness in NLP

Challenge:	Experimental results show that RLHF improves performance of Large Language Models . BT-based RMs struggle to distinguish between similar preference responses .
Approach:	They propose to enhance BT-based reward models by using an adaptive margin mechanism . they use semantic similarity and reward-predicted reward differences to adjust focus .
Outcome:	Experimental results show that the proposed method outperforms existing methods in both in-distribution and OOD settings.

WildReward: Learning Reward Models from In-the-Wild Human Interactions (2026.acl-long)

Copied to clipboard

Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Lei Hou, Juanzi Li

Challenge:	Prior work focused on collecting preference pairs, requiring substantial annotation efforts.
Approach:	They propose a pipeline to extract reliable human feedback from in-the-wild interactions . they propose to use WildChat as an interaction source to train the model .
Outcome:	The proposed model achieves comparable or even superior performance compared to conventional models with improved calibration and cross-sample consistency.