Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion (2024.emnlp-main)
Copied to clipboard
Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Eugene Choi, Bill Wu, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Azar, Olivier Pietquin, Matthieu Geist
| Challenge: | Reinforcement Learning (RL) is a method used to fine tune Large Language Models (LLMs) using a reward model trained from preference data to better align with human judgment. |
| Approach: | They propose a Reinforcement Learning (RL) algorithm that can estimate the optimal policy even from off-policy data. |
| Outcome: | The proposed algorithm can estimate the optimal policy even from off-policy data. |
Similar Papers
Reinforcement Learning for Large Language Models via Group Preference Reward Shaping (2025.emnlp-main)
Copied to clipboard
Huaisheng Zhu, Siyuan Xu, Hangfan Zhang, Teng Xiao, Zhimeng Guo, Shijie Zhou, Shuyue Hu, Vasant G. Honavar
| Challenge: | Existing methods for fine-tuning Large Language Models (LLMs) are expensive and sensitive to reward model quality. |
| Approach: | They propose a method that leverages preference-based comparisons rather than precise numerical rewards. |
| Outcome: | Experiments show that GPRS outperforms critic-model-free RL algorithms on RLHF and reasoning tasks. |
Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique for aligning large language models (LLMs) with human preferences. |
| Approach: | They propose a novel algorithm that uses multiple-gradient descent to optimize LLMs with diverse preferences to maximize trade-offs between objectives. |
| Outcome: | The proposed approach incorporates user preferences across different objectives and achieves Pareto solutions that better align with the user’s specific needs. |
Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for training large language models require additional annotations to adjust to shifted distributions. |
| Approach: | They propose an algorithm that allows LLMs and reward models to update alternatively via a min-max game to improve their alignment. |
| Outcome: | The proposed framework improves existing alignment baselines in terms of LLM helpfulness and harmlessness. |
Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling (2025.acl-long)
Copied to clipboard
| Challenge: | Existing safety alignment methods for Large Language Models (LLMs) face the distribution shift issue, which requires significant computational resources. |
| Approach: | They propose a framework that leverages the model’s intrinsic safety judgment capability to extract reward signals, which are then used to calculate label confidence for preference reordering. |
| Outcome: | The proposed framework improves safety performance while avoiding 300x computational overheads. |
RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Reinforcement learning with human feedback (RLHF) is widely employed to align large language models with user intent. |
| Approach: | They propose to combine rejection sampling and direct preference optimization to improve alignment with user intent by identifying pairs of contrastive samples from human annotator and alternative LLMs. |
| Outcome: | The proposed method outperforms existing methods including RS, PPO, and DPO in a limited resource environment. |
LIRE: listwise reward enhancement for preference alignment (2024.findings-acl)
Copied to clipboard
| Challenge: | prevailing approaches to preference alignment focus on pairwise comparisons, with limited exploration into multi-response scenarios. |
| Approach: | They propose a listwise reward enhancement approach that integrates offline rewards of multiple responses into a streamlined listwise framework. |
| Outcome: | The proposed approach outperforms existing methods on dialogue and summarization tasks with good transferability to out-of-distribution data. |
Don’t Forget Your Reward Values: Language Model Alignment via Value-based Calibration (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating large language models have been criticized for their complexity and instability. |
| Approach: | They propose a value-based calibration method to better align Large Language Models with human preferences. |
| Outcome: | The proposed method surpasses existing methods on AI assistant and summarization datasets, providing impressive generalizability, robustness, and diversity in different settings. |
Beyond Online Sampling: Bridging Offline-to-Online Alignment via Dynamic Data Transformation for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) eliminates complex reward modeling in aligning large language models with human preferences, but its online variant faces significant efficiency bottlenecks due to costly real-time preference sampling and the reward model annotation. |
| Approach: | They propose a framework that transforms static datasets into dynamically adaptive equivalents without the need for an explicit reward model. |
| Outcome: | The proposed approach matches or exceeds the performance of a fully online DPO. |
CLHA: A Simple Yet Effective Contrastive Learning Framework for Human Alignment (2024.lrec-main)
Copied to clipboard
Feiteng Fang, Liang Zhu, Xi Feng, Jinchang Hou, Qixuan Zhao, Chengming Li, Xiping Hu, Ruifeng Xu, Min Yang
| Challenge: | Large language models (LLMs) have attracted considerable attention from academic and industrial communities due to their outstanding performance in various natural language processing tasks. |
| Approach: | They propose a Contrastive Learning Framework for Human Alignment to evaluate the noise within the data and dynamically adjust the training process. |
| Outcome: | The proposed framework surpasses other algorithms in terms of reward model scores, automatic evaluations, and human assessments on the widely used dataset "Helpful and Harmless" |
Fine-Tuning Language Models with Reward Learning on Policy (2024.naacl-long)
Copied to clipboard
| Challenge: | Reinforcement learning from human feedback (RLHF) is an effective approach to align large language models (LLMs) to human preferences. |
| Approach: | They propose a framework that refines a reward model using policy samples to keep it on-distribution. |
| Outcome: | The proposed framework outperforms the state-of-the-art on three benchmark datasets showing that it can learn robust representations of policy samples. |