Tianning Chai, Chancharik Mitra, Brandon Huang, Gautam Rajendrakumar Gare, Zhiqiu Lin, Assaf Arbelle, Leonid Karlinsky, Rogerio Feris, Trevor Darrell, Deva Ramanan, Roei Herzig
| Challenge: | A common approach is to use reward models that enable reinforcement-learning post-training. |
| Approach: | They propose a method that steers LLM activations to align with few-shot preference data without finetuning. |
| Outcome: | The proposed method surpasses zero-shot, few-shot and voting-based benchmarks on reward hacking and noise signals. |
Similar Papers
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Reinforcement learning from human feedback (RLHF) is the primary method for aligning large language models with human preferences. |
| Approach: | They propose to train an Absolute-Rating Multi-Objective Reward Model with multi-dimensional absolute-rating data. |
| Outcome: | The proposed model outperforms the LLM-as-a-judge method on RewardBench . it achieves state-of-the-art performance on the benchmark . |
Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual human preference data are difficult to obtain at scale, making it challenging to extend this framework to diverse languages. |
| Approach: | They propose a method where a reward model is trained on preference data in one source language and applied to other target languages. |
| Outcome: | The proposed approach is effective under comprehensive evaluation settings, including human evaluation. |
APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that RLHF improves performance of Large Language Models . BT-based RMs struggle to distinguish between similar preference responses . |
| Approach: | They propose to enhance BT-based reward models by using an adaptive margin mechanism . they use semantic similarity and reward-predicted reward differences to adjust focus . |
| Outcome: | Experimental results show that the proposed method outperforms existing methods in both in-distribution and OOD settings. |
RewardBench: Evaluating Reward Models for Language Modeling (2025.findings-naacl)
Copied to clipboard
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi
| Challenge: | Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models. |
| Approach: | They present a benchmark dataset and code-base for evaluation of reward models . they use prompt-chosen-rejected trios to benchmark how they perform on queries . |
| Outcome: | The proposed dataset compares RMs with other models on a set of questions. |
Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | Existing failure discovery methods rely on prior knowledge of preference attributes . Existing methods do not scale to new models or data. |
| Approach: | They propose a preference distribution agnostic procedure that uses the reward model itself to guide controlled decoding toward mis specified responses while preserving the underlying preference class. |
| Outcome: | The proposed procedure improves robustness without degrading reward quality across models. |
Debiasing Reward Models via Causally Motivated Inference-Time Intervention (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches for mitigating spurious features in RMs focus on response length . Existing methods focus on RM activation, resulting in performance trade-offs . |
| Approach: | They propose a method that uses neurons to suppress spurious features in RMs at inference time. |
| Outcome: | The proposed method reduces sensitivity to spurious features without inducing performance trade-offs on RM benchmarks. |
RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment (2025.findings-acl)
Copied to clipboard
Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, Jun Zhao
| Challenge: | Existing retrieval augmented language models often overlook effective alignment with human preferences. |
| Approach: | They propose a benchmark to evaluate RMs in retrieval augmented language models . they incorporate 18 RAG subsets, six retrievers, and 24 RALMs to increase diversity . |
| Outcome: | The proposed benchmark combines 18 RAG subsets, six retrievers, and 24 RALMs to increase diversity of data sources. |
PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | Existing reward models lack generative and reasoning capabilities, resulting in poor performance. |
| Approach: | They propose a reward-aware task-adaptive reward model that enables pointwise training using readily available pairwise data via a novel Preference-Aware Reward mechanism. |
| Outcome: | The proposed reward model achieves an average relative improvement of 8.7% over the base models on RewardBench and RMBench. |
WildReward: Learning Reward Models from In-the-Wild Human Interactions (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work focused on collecting preference pairs, requiring substantial annotation efforts. |
| Approach: | They propose a pipeline to extract reliable human feedback from in-the-wild interactions . they propose to use WildChat as an interaction source to train the model . |
| Outcome: | The proposed model achieves comparable or even superior performance compared to conventional models with improved calibration and cross-sample consistency. |
Reward Model Perspectives: Whose Opinions Do Reward Models Reward? (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that reward models are poorly aligned with demographic groups and can reward harmful stereotypes. |
| Approach: | They propose a framework for measuring the alignment of opinions captured by RMs . they also investigate the extent to which RM's demonstrate sociodemographic biases a . |
| Outcome: | The proposed framework measures the alignment of opinions captured by RMs . it shows that RM models are poorly aligned with several demographic groups . the findings highlight the need for more careful consideration of RM behavior in model alignment . |