PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | Existing reward models lack generative and reasoning capabilities, resulting in poor performance. |
| Approach: | They propose a reward-aware task-adaptive reward model that enables pointwise training using readily available pairwise data via a novel Preference-Aware Reward mechanism. |
| Outcome: | The proposed reward model achieves an average relative improvement of 8.7% over the base models on RewardBench and RMBench. |
Similar Papers
APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that RLHF improves performance of Large Language Models . BT-based RMs struggle to distinguish between similar preference responses . |
| Approach: | They propose to enhance BT-based reward models by using an adaptive margin mechanism . they use semantic similarity and reward-predicted reward differences to adjust focus . |
| Outcome: | Experimental results show that the proposed method outperforms existing methods in both in-distribution and OOD settings. |
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Reinforcement learning from human feedback (RLHF) is the primary method for aligning large language models with human preferences. |
| Approach: | They propose to train an Absolute-Rating Multi-Objective Reward Model with multi-dimensional absolute-rating data. |
| Outcome: | The proposed model outperforms the LLM-as-a-judge method on RewardBench . it achieves state-of-the-art performance on the benchmark . |
ToolRM: Towards Agentic Tool-Use Reward Modeling (2026.findings-acl)
Copied to clipboard
Renhao Li, Jianhong Tu, Yang Su, Yantao Liu, Fei Huang, Hamid Alinejad-Rokny, Derek F. Wong, Junyang Lin, Min Yang
| Challenge: | lack of reliable reward models for tool-use tasks has limited progress toward agentic AI . recent advances in agentic artificial intelligence are driven by tool-using capabilities of large language models. |
| Approach: | They propose a pipeline that constructs pairwise preference data using rule-based scoring and multidimensional sampling to build lightweight reward models. |
| Outcome: | The proposed model outperforms existing models on tool calling tasks with higher accuracy. |
ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework (2026.acl-long)
Copied to clipboard
Kai Qin, Liangxin Liu, Yu Liang, Longzheng Wang, null Wangyan, Zhang Yueyang, Long Xia, Zhiyuan Sun, Houde Liu, Daiting Shi
| Challenge: | Existing methods for generating reward models focus on outcome-level supervision, neglecting analytical process quality, which constrains their potential. |
| Approach: | They propose a novel reward model that leverages self-reflection to assess analytical quality and enhance preference modeling. |
| Outcome: | The proposed model improves performance on four benchmarks and significantly mitigates positional bias. |
RewardBench: Evaluating Reward Models for Language Modeling (2025.findings-naacl)
Copied to clipboard
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi
| Challenge: | Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models. |
| Approach: | They present a benchmark dataset and code-base for evaluation of reward models . they use prompt-chosen-rejected trios to benchmark how they perform on queries . |
| Outcome: | The proposed dataset compares RMs with other models on a set of questions. |
ARF-RLHF: Adaptive Reward-Following for RLHF through Emotion-Driven Self-Supervision and Trace-Biased Dynamic Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | prevailing RLHF methods such as PPO and DPO depend on large-scale binary preference annotations. |
| Approach: | They propose a method which converts natural feedback into continuous preference trajectories and optimizes them using the novel TraceBias algorithm. |
| Outcome: | The proposed approach outperforms PPO and DPO in a variety of domains and improves alignment by up to 7.6% across diverse LLMs and preference domains. |
DUAL RM: Beyond Rule-based Preference Reward Modeling via Meta-Reward (2026.acl-long)
Copied to clipboard
Xiaobo Liang, Wanfu Wang, Qipeng Huang, Yuyang Ding, Zecheng Tang, Yixin Ji, Qianben Chen, Zhe Zhao, Kehai Chen, Juntao Li, Min Zhang
| Challenge: | Existing preference-based reward modeling methods face a recursive dependency where each verifier requires a meta-verifier, leading to continuous and costly dependence on human annotation. |
| Approach: | They propose a dual RM that couples discriminative and generative reward models under a non-parametric meta-reward. |
| Outcome: | The proposed model achieves strong performance across major preference benchmarks and even when trained exclusively on language modality, it exhibits robust cross-modal transfer on Omni-RewardBench. |
RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results demonstrate the superiority of our approach to aligning large language models with human preferences. |
| Approach: | They propose a method that evaluates and assigns specific credit to each token using an off-the-shelf reward model. |
| Outcome: | The proposed method evaluates and assigns specific credit to each token using an off-the-shelf reward model. |
OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment (2026.acl-long)
Copied to clipboard
| Challenge: | Existing reward models rely on scalar or pairwise judgments that fail to capture multifaceted nature of human preferences. |
| Approach: | They propose a rubric-based reward model that uses a large collection of prompt, rubric pairs to generate a scalar score or preference label for each response. |
| Outcome: | The proposed model surpasses strong size-matched baselines by 8.4% across multiple benchmarks. |
PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing approaches to align large language models with human preferences are limited by their large-scale annotation and prone to reward overoptimization. |
| Approach: | They propose a training paradigm that integrates three complementary strategies to address these challenges by reformulating question–answer pairs into preference-task instructions, averaging the rewards aggregated from diverse preference- task instructions for each sample, and a balancing outputs from the value head under different dropout rates. |
| Outcome: | Experiments on public datasets show that PIRA improves performance considerably, enhances generalization, and effectively mitigates reward overoptimization. |