RewardBench: Evaluating Reward Models for Language Modeling

Challenge:	Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models.
Approach:	They present a benchmark dataset and code-base for evaluation of reward models . they use prompt-chosen-rejected trios to benchmark how they perform on queries .
Outcome:	The proposed dataset compares RMs with other models on a set of questions.

Similar Papers

Overview

Challenge

Approach

Outcome

M-RewardBench: Evaluating Reward Models in Multilingual Settings (2025.acl-long)

Copied to clipboard

Srishti Gureja, Lester James Validad Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Triandi Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, Marzieh Fadaee

Multilinguality

Challenge:	Reward models (RMs) are primarily trained and evaluated in English and their capabilities in multilingual settings remain understudied.
Approach:	They construct a multilingual RM evaluation benchmark that tests the chat, safety, reasoning, and translation capabilities of RMs in 23 languages.
Outcome:	The proposed model performs better for high-resource languages and improves with translation quality.

A Systematic Analysis of Base Model Choice for Reward Modeling (2025.emnlp-main)

Copied to clipboard

Kian Ahrabian, Pegah Jandaghi, Negar Mokhberian, Sai Praneeth Karimireddy, Jay Pujara

Challenge:	Reinforcement learning from human feedback (RLHF) and reward modeling are key to training powerful large language models (LLMs).
Approach:	They propose to combine RLHF and reward modeling to boost model selection . they also demonstrate that a small set of benchmarks could be combined to boost the model selection.
Outcome:	The results show that the model selection can be improved by up to 14% compared to the most common (default) choice.

Reward Model Perspectives: Whose Opinions Do Reward Models Reward? (2025.emnlp-main)

Copied to clipboard

null Elle

Language Models

Challenge:	a recent study shows that reward models are poorly aligned with demographic groups and can reward harmful stereotypes.
Approach:	They propose a framework for measuring the alignment of opinions captured by RMs . they also investigate the extent to which RM's demonstrate sociodemographic biases a .
Outcome:	The proposed framework measures the alignment of opinions captured by RMs . it shows that RM models are poorly aligned with several demographic groups . the findings highlight the need for more careful consideration of RM behavior in model alignment .

Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling (2026.acl-long)

Copied to clipboard

Jiaxuan Wang, Yulan Hu, Wenjin Yang, Zheng Pan, Xin Li, Lan-Zhe Guo

Challenge:	Large Language Models (LLMs) evolve into agentic systems capable of autonomous tool invocation and complex reasoning.
Approach:	They propose a trajectory-level preference benchmark to evaluate judges' ability to distinguish preferred versus distractor agent trajectories in tool-integrated environments.
Outcome:	The proposed benchmark evaluates how well judges distinguish preferred versus distractor agent trajectories in complex tool-using scenarios.

Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization (2025.acl-long)

Copied to clipboard

Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Dongha Lee, Jinyoung Yeo

Challenge:	Existing benchmarks for reward models show a weak correlation with performance of optimized policies . existing benchmarks do not accurately assess the true capabilities of reward models .
Approach:	They explore how reward overoptimization captures how well a reward model aligns with human preferences and the dynamics of the learning signal it provides to the policy.
Outcome:	The proposed benchmarks show that reward overoptimization is a weak factor . the high correlation with degree of overoptimalization leads to lower correlation with downstream performance .

RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment (2025.findings-acl)

Copied to clipboard

Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, Jun Zhao

Information Retrieval

Challenge:	Existing retrieval augmented language models often overlook effective alignment with human preferences.
Approach:	They propose a benchmark to evaluate RMs in retrieval augmented language models . they incorporate 18 RAG subsets, six retrievers, and 24 RALMs to increase diversity .
Outcome:	The proposed benchmark combines 18 RAG subsets, six retrievers, and 24 RALMs to increase diversity of data sources.

Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024.findings-emnlp)

Copied to clipboard

Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, Tong Zhang

Explainability & Interpretability in NLP

Challenge:	Reinforcement learning from human feedback (RLHF) is the primary method for aligning large language models with human preferences.
Approach:	They propose to train an Absolute-Rating Multi-Objective Reward Model with multi-dimensional absolute-rating data.
Outcome:	The proposed model outperforms the LLM-as-a-judge method on RewardBench . it achieves state-of-the-art performance on the benchmark .

reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs (2025.emnlp-main)

Copied to clipboard

Zhaofeng Wu, Michihiro Yasunaga, Andrew Cohen, Yoon Kim, Asli Celikyilmaz, Marjan Ghazvininejad

Robustness in NLP

Challenge:	Existing reward models have a high performance on benchmarks, but performance degradation is often due to overfitting.
Approach:	They propose to explicitly train reward models to assign similar scores to paraphrases to improve their robustness.
Outcome:	The proposed model reduces degradation by half for the Chat Hard subset in RewardBench.

WildReward: Learning Reward Models from In-the-Wild Human Interactions (2026.acl-long)

Copied to clipboard

Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Lei Hou, Juanzi Li

Challenge:	Prior work focused on collecting preference pairs, requiring substantial annotation efforts.
Approach:	They propose a pipeline to extract reliable human feedback from in-the-wild interactions . they propose to use WildChat as an interaction source to train the model .
Outcome:	The proposed model achieves comparable or even superior performance compared to conventional models with improved calibration and cross-sample consistency.

ToolRM: Towards Agentic Tool-Use Reward Modeling (2026.findings-acl)

Copied to clipboard

Renhao Li, Jianhong Tu, Yang Su, Yantao Liu, Fei Huang, Hamid Alinejad-Rokny, Derek F. Wong, Junyang Lin, Min Yang

Challenge:	lack of reliable reward models for tool-use tasks has limited progress toward agentic AI . recent advances in agentic artificial intelligence are driven by tool-using capabilities of large language models.
Approach:	They propose a pipeline that constructs pairwise preference data using rule-based scoring and multidimensional sampling to build lightweight reward models.
Outcome:	The proposed model outperforms existing models on tool calling tasks with higher accuracy.