RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have significantly enhanced the capabilities in natural language processing. |
| Approach: | They propose a method to poison large language models by using annotators to rank a set of collected responses to generate longer tokens. |
| Outcome: | The proposed method can generate longer tokens without harming the original safety alignment performance. |
Similar Papers
RLHF Algorithms Ranked: An Extensive Evaluation Across Diverse Tasks, Rewards, and Hyperparameters (2025.emnlp-industry)
Copied to clipboard
Lucas Spangher, Rama Kumar Pasumarthi, Nick Masiewicki, William F. Arnold, Aditi Kaushal, Dale Johnson, Peter Grabowski, Eugene Ie
| Challenge: | Proximal Policy Optimization (PPO) has fallen out of favor for Large Language Models (LLMs), but its complexity and inefficiency have spurred the investigation of simpler alternatives. |
| Approach: | They evaluate 17 RLHF algorithms on two benchmarks, OpenAI’s TL;DR Summarization and Anthropic’s Helpfulness / Harmlessness. |
| Outcome: | The proposed methods are based on OpenAI’s TL;DR Summarization and Anthropic’s Helpfulness / Harmlessness benchmarks with two different reward models and a Rules based reward model. |
RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results demonstrate the superiority of our approach to aligning large language models with human preferences. |
| Approach: | They propose a method that evaluates and assigns specific credit to each token using an off-the-shelf reward model. |
| Outcome: | The proposed method evaluates and assigns specific credit to each token using an off-the-shelf reward model. |
Improving Discriminative Capability of Reward Models in RLHF Using Contrastive Learning (2024.emnlp-main)
Copied to clipboard
Lu Chen, Rui Zheng, Binghai Wang, Senjie Jin, Caishuang Huang, Junjie Ye, Zhihao Zhang, Yuhao Zhou, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Current methods rely on ranking losses to teach reward model to assess preferences, but they are susceptible to noise and ambiguous data, often failing to deeply understand human intentions. |
| Approach: | They propose a method that incorporates contrastive learning into the reward modeling process to enhance generalization and stabilize the reinforcement learning training process. |
| Outcome: | The proposed method enhances generalization of the reward model, stabilizes the reinforcement learning training process, and improves the final alignment with human preferences. |
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Reinforcement learning from human feedback (RLHF) is the primary method for aligning large language models with human preferences. |
| Approach: | They propose to train an Absolute-Rating Multi-Objective Reward Model with multi-dimensional absolute-rating data. |
| Outcome: | The proposed model outperforms the LLM-as-a-judge method on RewardBench . it achieves state-of-the-art performance on the benchmark . |
Towards Reward Fairness in RLHF: From a Resource Allocation Perspective (2025.acl-long)
Copied to clipboard
| Challenge: | if rewards are imperfect, they can adversely affect the alignment of large language models (LLMs). |
| Approach: | They propose a bias-agnostic method to address the issue of reward unfairness from a resource allocation perspective without specifically designing for each type of bias . they apply methods Fairness Regularization and Fairness Coefficient to achieve fairness in rewards. |
| Outcome: | The proposed method achieves fairness in rewards while minimizing biases . it can be applied to verification and reinforcement learning scenarios . |
Understanding Impact of Human Feedback via Influence Functions (2025.acl-long)
Copied to clipboard
| Challenge: | In reinforcement learning from human feedback, human feedback can be noisy, inconsistent or biased . this variability can lead to misaligned reward signals, potentially causing unintended side effects . |
| Approach: | They propose an approximation method that measures the impact of human feedback on the performance of reward models. |
| Outcome: | The proposed method detects common labeler biases in human feedback datasets and guides labelers in refining their strategies to better align with expert feedback. |
Aligning to What? Limits to RLHF Based Alignment (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies on RLHF and covert and overt biases in large language models are unclear . et al. analyzed off-the-shelf language models to evaluate their overt and cover racial biase . |
| Approach: | They evaluate the relationship between reinforcement learning from human feedback and biases in large language models. |
| Outcome: | The proposed approach can be used to mitigat covert biases, the authors show . they found that the RLHF approach calcifies model biase . |
Beyond Excess and Deficiency: Adaptive Length Bias Mitigation in Reward Models for RLHF (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing efforts to mitigate length bias in reward models have decreased accuracy . achieving an automatic proxy that perfectly replicates human judgment is challenging in practice. |
| Approach: | They propose an adaptive approach that dynamically adjusts the influence of response length in reward evaluations according to the context of the query. |
| Outcome: | The proposed approach reduces unnecessary verbosity while improving overall response quality. |
Removing RLHF Protections in GPT-4 via Fine-Tuning (2024.naacl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) have increased in their capabilities, which increases their potential for dual use. |
| Approach: | They show that fine-tuning can remove RLHFprotections with as few as 340 examples and a 95% success rate. |
| Outcome: | The proposed method removes RLHFprotections with as few as 340 examples and a 95% success rate on non-censored outputs. |
Reward Difference Optimization For Sample Reweighting In Offline RLHF (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are becoming more capable, but their maximum likelihood objective for the next token prediction falls short in capturing such crucial human values. |
| Approach: | They propose a reward difference prediction method that uses reward difference coefficients to reweigh sample pairs in offline RLHF and a difference model that considers rich interactions between a pair of responses. |
| Outcome: | The proposed method is effective in both automatic metrics and human evaluation. |