Zhuohao Yu, Jiali Zeng, Weizheng Gu, Mengyuan Sun, Yidong Wang, Fandong Meng, Jie Zhou, Shikun Zhang, Wei Ye
| Challenge: | JODP optimizes policies on fixed training inputs, limiting the diversity of learning signals. |
| Approach: | They propose a framework where policy generates improved variants of training problems to enhance its own learning. |
| Outcome: | The proposed framework improves on safety alignment tasks by allowing 4B models to reach 8B model performance with less than 1% additional computational overhead. |
Similar Papers
RLHF Algorithms Ranked: An Extensive Evaluation Across Diverse Tasks, Rewards, and Hyperparameters (2025.emnlp-industry)
Copied to clipboard
Lucas Spangher, Rama Kumar Pasumarthi, Nick Masiewicki, William F. Arnold, Aditi Kaushal, Dale Johnson, Peter Grabowski, Eugene Ie
| Challenge: | Proximal Policy Optimization (PPO) has fallen out of favor for Large Language Models (LLMs), but its complexity and inefficiency have spurred the investigation of simpler alternatives. |
| Approach: | They evaluate 17 RLHF algorithms on two benchmarks, OpenAI’s TL;DR Summarization and Anthropic’s Helpfulness / Harmlessness. |
| Outcome: | The proposed methods are based on OpenAI’s TL;DR Summarization and Anthropic’s Helpfulness / Harmlessness benchmarks with two different reward models and a Rules based reward model. |
WPO: Enhancing RLHF with Weighted Preference Optimization (2024.emnlp-main)
Copied to clipboard
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, Chenguang Zhu
| Challenge: | Off-policy preference optimization suffers from a distributional gap between the policy used for data collection and the target policy, leading to suboptimal optimization. |
| Approach: | They propose a method to simulate on-policy learning with off-police preference data. |
| Outcome: | The proposed method outperforms Direct Preference Optimization (DPO) by up to 5.6% on Alpaca Eval 2 and MT-bench. |
Fine-Tuning Language Models with Reward Learning on Policy (2024.naacl-long)
Copied to clipboard
| Challenge: | Reinforcement learning from human feedback (RLHF) is an effective approach to align large language models (LLMs) to human preferences. |
| Approach: | They propose a framework that refines a reward model using policy samples to keep it on-distribution. |
| Outcome: | The proposed framework outperforms the state-of-the-art on three benchmark datasets showing that it can learn robust representations of policy samples. |
Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs (2024.acl-long)
Copied to clipboard
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker
| Challenge: | Proximal Policy Optimization (PPO) is used for RLHF but requires high computational cost and sensitive hyperparameter tuning. |
| Approach: | They propose to use Proximal Policy Optimization to align large language models to human preferences. |
| Outcome: | The proposed method preserves and even increases performance while preserving the motivational principles that led to the development of PPO. |
trlX: A Framework for Large Scale Reinforcement Learning from Human Feedback (2023.emnlp-main)
Copied to clipboard
Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, Louis Castricato
| Challenge: | Current RLHF paradigms rely on Proximal Policy Optimization (PPO), which quickly becomes a challenge to implement and scale up to large architectures. |
| Approach: | They propose an open-source framework for reinforcement learning from human feedback . it allows for offline fine-tuning of large language models . |
| Outcome: | The framework can be used to fine-tune models up to and exceeding 70 billion parameters. |
Efficient Hyperparameter Optimization for LLM Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing hyperparameter optimization methods are inefficient in reinforcement learning due to model scale and resource-intensive training cycles. |
| Approach: | They propose a hyperparameter optimization method that adapts both model size and training budget as fidelity. |
| Outcome: | The proposed method significantly improves the computational efficiency of each trial (up to 14.9) over existing HPO methods. |
Don’t Tell the Answer, Truly Guide the Reasoning During RL Rollouts (2026.findings-acl)
Copied to clipboard
Xinyi Wang, Jinyi Han, Zishang Jiang, Tingyun li, Jiaqing Liang, Sihang Jiang, Zhaoqian Dai, Ma Shuguang, Fei Yu, Yanghua Xiao
| Challenge: | Existing methods such as GRPO often break down when task difficulty exceeds the model’s capacity, resulting in sparse rewards and inefficient training. |
| Approach: | They propose to measure the compatibility between external guidance and a model's intrinsic policy by introducing an adaptive framework to enhance reasoning performance while explicitly preserving high Affinity. |
| Outcome: | The proposed framework outperforms baseline models while maintaining high Affinity. |
[CASPI] Causal-aware Safe Policy Improvement for Task-oriented Dialogue (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in off-policy reinforcement learning methods that use offline data as against a simulator have proven to be sample efficient. |
| Approach: | They propose a batch-RL framework for ToD policy learning: Causal-aware Safe Policy Improvement (CASPI) that uses a mechanism to learn fine-grained reward that captures intention behind human response and offers guarantee on dialogue policy’s performance against a baseline. |
| Outcome: | The proposed framework outperforms the current state of the art on an end-to-end dialogue task using a multiwoz2.0 dataset. |
Towards Pareto-Efficient RLHF: Paying Attention to a Few High-Reward Samples with Reward Dropout (2024.findings-emnlp)
Copied to clipboard
| Challenge: | RLHF is a bi-objective problem that has the nature of a Pareto optimization . reward dropout is generalizable and most effective with non-pretrained target models . |
| Approach: | They propose a method that guarantees a Pareto improvement by leveraging reinforcement learning to fine-tune language models. |
| Outcome: | The proposed method guarantees a Pareto improvement on two benchmark datasets . it is generalizable and most effective with non-pretrained target models, saving the effort of pretraining. |
Mutual-Taught for Co-adapting Policy and Reward Models (2025.acl-long)
Copied to clipboard
Tianyuan Shi, Canbin Huang, Fanqi Wan, Longguang Zhong, Ziyi Yang, Weizhou Shen, Xiaojun Quan, Ming Yan
| Challenge: | Experimental results show that this iterative approach leads to consistent improvements in both the policy model and reward model. |
| Approach: | They propose a method that iteratively improves both the policy model and reward model without requiring additional human annotation. |
| Outcome: | The proposed method improves both the policy model and reward model without human annotation. |