LiPO: Listwise Preference Optimization through Learning-to-Rank (2025.naacl-long)
Copied to clipboard
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, Peter J Liu, Xuanhui Wang
| Challenge: | Recent work on language models with curated feedback provides promising alternatives to RLHF . multiple responses can be ranked by reward models or AI feedback, but there is no study on directly fitting upon a list of responses. |
| Approach: | They propose a method that aligns language models with curated human feedback . they propose SLiC and DPO as promising alternatives to traditional RLHF . |
| Outcome: | The proposed method outperforms DPO and SLiC on several preference alignment tasks with curated and real rankwise preference data. |
Similar Papers
Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have focused on replacing the reward model in Reinforcement Learning with Human Feedback (RLHF) methods for Large Language Models (LLMs). |
| Approach: | They propose a self-supervised preference optimization framework that replaces the reward model with a preference loss and alignment loss to improve LLMs' ability to understand human preferences. |
| Outcome: | The proposed framework can be integrated with existing preference optimization methods and significantly boost their performance. |
K-order Ranking Preference Optimization for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing list-wise methods focus on optimizing list ranking consistency for LLMs to improve ranking abilities. |
| Approach: | They propose to extend the Plackett-Luce model to accommodate top-K ranking by extending the DPO’s Plact-Lucer model to dynamically determine appropriate K for different samples. |
| Outcome: | The proposed model can be extended to accommodate top-K ranking and improve training efficiency. |
Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique for aligning large language models (LLMs) with human preferences. |
| Approach: | They propose a novel algorithm that uses multiple-gradient descent to optimize LLMs with diverse preferences to maximize trade-offs between objectives. |
| Outcome: | The proposed approach incorporates user preferences across different objectives and achieves Pareto solutions that better align with the user’s specific needs. |
LIRE: listwise reward enhancement for preference alignment (2024.findings-acl)
Copied to clipboard
| Challenge: | prevailing approaches to preference alignment focus on pairwise comparisons, with limited exploration into multi-response scenarios. |
| Approach: | They propose a listwise reward enhancement approach that integrates offline rewards of multiple responses into a streamlined listwise framework. |
| Outcome: | The proposed approach outperforms existing methods on dialogue and summarization tasks with good transferability to out-of-distribution data. |
RLHF Algorithms Ranked: An Extensive Evaluation Across Diverse Tasks, Rewards, and Hyperparameters (2025.emnlp-industry)
Copied to clipboard
Lucas Spangher, Rama Kumar Pasumarthi, Nick Masiewicki, William F. Arnold, Aditi Kaushal, Dale Johnson, Peter Grabowski, Eugene Ie
| Challenge: | Proximal Policy Optimization (PPO) has fallen out of favor for Large Language Models (LLMs), but its complexity and inefficiency have spurred the investigation of simpler alternatives. |
| Approach: | They evaluate 17 RLHF algorithms on two benchmarks, OpenAI’s TL;DR Summarization and Anthropic’s Helpfulness / Harmlessness. |
| Outcome: | The proposed methods are based on OpenAI’s TL;DR Summarization and Anthropic’s Helpfulness / Harmlessness benchmarks with two different reward models and a Rules based reward model. |
Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs (2024.acl-long)
Copied to clipboard
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker
| Challenge: | Proximal Policy Optimization (PPO) is used for RLHF but requires high computational cost and sensitive hyperparameter tuning. |
| Approach: | They propose to use Proximal Policy Optimization to align large language models to human preferences. |
| Outcome: | The proposed method preserves and even increases performance while preserving the motivational principles that led to the development of PPO. |
Comparing Bad Apples to Good Oranges Aligning Large Language Models via Joint Preference Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that acquiring human preferences by comparing generations is not effective for large language models. |
| Approach: | They propose a preference optimization objective that elicits preferences jointly over the instruction-response pairs. |
| Outcome: | The proposed approach outperforms prior preference optimizations by 5.2% and 3.3% in summarization and open-ended dialogue datasets. |
LPOI: Listwise Preference Optimization for Vision Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for aligning large VLMs with human preferences often overfit to textual information or exacerbate hallucinations. |
| Approach: | They propose an object-aware listwise preference optimization for reducing hallucinations in VLMs . they mask a critical object in an image and interpolate the masked region to form more complete images . |
| Outcome: | The proposed method outperforms existing methods in reducing hallucinations and enhancing performance on MMHalBench, AMBER, and Object HalBench. |
Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat (2025.acl-long)
Copied to clipboard
| Challenge: | Evaluating large language models (LLMs) is a complex task. Pairwise ranking has emerged as state-of-the-art method to evaluate human preferences. |
| Approach: | They propose to use pairwise ranking to evaluate human preferences . they propose to evaluate the robustness of ranking algorithms in LLMs . |
| Outcome: | The proposed methods are based on the principles of effective ranking and the robustness of several ranking algorithms in the context of LLMs. |
Permutative Preference Alignment from Listwise Ranking of Human Judgments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to align Large Language Models with human preferences are based on the Bradley-Terry model, but when multiple responses are available, the B-T model fails to guarantee an accurate list ranking of the responses. |
| Approach: | They propose an offline listwise approach that incorporates the Normalized Discounted Cumulative Gain (NDCG) as an alternative training objective for LLM alignment. |
| Outcome: | The proposed approach outperforms existing pairwise and listwise methods on evaluation sets and general benchmarks such as AlpacaEval. |