Debiasing Reward Models via Causally Motivated Inference-Time Intervention (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches for mitigating spurious features in RMs focus on response length . Existing methods focus on RM activation, resulting in performance trade-offs . |
| Approach: | They propose a method that uses neurons to suppress spurious features in RMs at inference time. |
| Outcome: | The proposed method reduces sensitivity to spurious features without inducing performance trade-offs on RM benchmarks. |
Similar Papers
Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | Existing failure discovery methods rely on prior knowledge of preference attributes . Existing methods do not scale to new models or data. |
| Approach: | They propose a preference distribution agnostic procedure that uses the reward model itself to guide controlled decoding toward mis specified responses while preserving the underlying preference class. |
| Outcome: | The proposed procedure improves robustness without degrading reward quality across models. |
Removing Prompt-template Bias in Reinforcement Learning from Human Feedback (2025.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) has shown promise for enhancing pre-trained large language models to generate responses that align with human preferences and societal values. |
| Approach: | They propose a method to estimate prompt-template bias term during reward modeling and use it to calibrate reward scores. |
| Outcome: | The proposed method can be flexibly combined with existing algorithms of removing length bias, leading to a further improvement in the aspect of enhancing the quality of generated responses. |
BiasFilter: An Inference-Time Debiasing Framework for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for debiasing large language models incur high human and computational costs and are limited in their effectiveness. |
| Approach: | They propose a model-agnostic, inference-time debiasing framework that enforces fairness by filtering generation outputs in real time. |
| Outcome: | The proposed framework mitigates social bias across a range of LLMs while preserving overall generation quality. |
Aligning as Debiasing: Causality-Aware Alignment via Reinforcement Learning with Interventional Feedback (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to reduce LLMs' biased outputs rely on reward signals from current model outputs without considering the source of biases. |
| Approach: | They propose to leverage the reward model in RL alignment as an instrumental variable to perform causal intervention on LLMs. |
| Outcome: | The proposed method reduces biases by using human feedback to fine tune LLMs to human values. |
Bias Fitting to Mitigate Length Bias of Reward Model in RLHF (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to tackling length bias are limited by their complexity or lack of a linear length-reward relation. |
| Approach: | They propose a framework that learns and corrects underlying bias patterns by fitting a length-reward relationship into a reward model. |
| Outcome: | The proposed framework improves length-controlled win rate and reduces verbosity without compromising performance. |
Activation Reward Models for Few-Shot Model Alignment (2026.findings-acl)
Copied to clipboard
Tianning Chai, Chancharik Mitra, Brandon Huang, Gautam Rajendrakumar Gare, Zhiqiu Lin, Assaf Arbelle, Leonid Karlinsky, Rogerio Feris, Trevor Darrell, Deva Ramanan, Roei Herzig
| Challenge: | A common approach is to use reward models that enable reinforcement-learning post-training. |
| Approach: | They propose a method that steers LLM activations to align with few-shot preference data without finetuning. |
| Outcome: | The proposed method surpasses zero-shot, few-shot and voting-based benchmarks on reward hacking and noise signals. |
APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that RLHF improves performance of Large Language Models . BT-based RMs struggle to distinguish between similar preference responses . |
| Approach: | They propose to enhance BT-based reward models by using an adaptive margin mechanism . they use semantic similarity and reward-predicted reward differences to adjust focus . |
| Outcome: | Experimental results show that the proposed method outperforms existing methods in both in-distribution and OOD settings. |
DARM: Distribution-Aware Reward Modeling by Alleviating Biases from Low Preference-Context Dependency Data (2026.acl-long)
Copied to clipboard
Shaofan Liu, Guoqiang Zhang, Shihan Dou, Huiyuan Zheng, Yiming Zhou, Junjie Ye, Shaowen Wang, Shichun Liu, Jiazheng Zhang, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing methods for training reward models are vulnerable to context neglect and degraded accuracy. |
| Approach: | They propose distribution-aware reward modeling that augments the RM objective with a conditional mutual information regularizer that maximizes context and the predicted reward conditioned on the response. |
| Outcome: | The proposed model improves performance in RLHF and improves accuracy in other settings. |
Beyond Excess and Deficiency: Adaptive Length Bias Mitigation in Reward Models for RLHF (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing efforts to mitigate length bias in reward models have decreased accuracy . achieving an automatic proxy that perfectly replicates human judgment is challenging in practice. |
| Approach: | They propose an adaptive approach that dynamically adjusts the influence of response length in reward evaluations according to the context of the query. |
| Outcome: | The proposed approach reduces unnecessary verbosity while improving overall response quality. |
Inference-Time Selective Debiasing to Enhance Fairness in Text Classification Models (2025.naacl-short)
Copied to clipboard
| Challenge: | Several studies have investigated and promoted fairness, and a variety of definitions have been proposed to address this problem. |
| Approach: | They propose a selective debiasing method that removes bias from model predictions instead of discarding them at inference time. |
| Outcome: | The proposed method achieves better results than standard uncertainty quantification methods on text classification datasets with encoder-based classification models. |