Adaptive Preference Optimization with Uncertainty-aware Utility Anchor (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Offline preference optimization methods are efficient for large language models (LLMs) alignment. |
| Approach: | They propose an offline preference optimization framework that estimates uncertainties from preference data . the method enables training even in scenarios where the data is unpaired . |
| Outcome: | The proposed method enables training even in scenarios where the data is unpaired . |
Similar Papers
Beyond Online Sampling: Bridging Offline-to-Online Alignment via Dynamic Data Transformation for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) eliminates complex reward modeling in aligning large language models with human preferences, but its online variant faces significant efficiency bottlenecks due to costly real-time preference sampling and the reward model annotation. |
| Approach: | They propose a framework that transforms static datasets into dynamically adaptive equivalents without the need for an explicit reward model. |
| Outcome: | The proposed approach matches or exceeds the performance of a fully online DPO. |
Robust Preference Optimization via Dynamic Target Margins (2025.findings-acl)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) is an efficient method for ensuring safety and reliability in practical applications. |
| Approach: | They propose a dynamic target margin preference optimization algorithm that adjusts reward margins at the pairwise level. |
| Outcome: | The proposed method achieves an average 4.4% improvement over baselines, setting new benchmarks for state-of-the-art performance. |
Expectation Preference Optimization: Reliable Preference Estimation for Improving the Reasoning Capability of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Pairwise preference optimization is used to improve supervised fine-tuning performance of large language models. |
| Approach: | They propose an algorithm that takes pairs of sample groups instead of single samples for preference learning. |
| Outcome: | The proposed algorithm outperforms baseline methods on reasoning benchmarks. |
BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to align Large Language Models with human preferences fail to maintain general knowledge and alignment when faced with personalized preferences. |
| Approach: | They propose a method that utilizes the initial responses of the reference model to mitigate forgetting while accommodating personalized alignment. |
| Outcome: | The proposed approach mitigates forgetting while accommodating personalized alignment while preserving global knowledge and general alignment. |
InfoPO: On Mutual Information Maximization for Large Language Model Alignment (2025.naacl-long)
Copied to clipboard
Teng Xiao, Zhen Ge, Sujay Sanghavi, Tian Wang, Julian Katz-Samuels, Marc Versage, Qingjun Cui, Trishul Chilimbi
| Challenge: | Recent studies have shown that direct preference optimization and its variants can be useful for fine-tuning large language models with human preferences data. |
| Approach: | They propose a preference fine-tuning algorithm that effectively and efficiently aligns large language models using preference data. |
| Outcome: | Extensive experiments show that the proposed algorithm outperforms established baselines on reasoning tasks. |
Uncertainty-Aware Iterative Preference Optimization for Enhanced LLM Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for enhancing the performance of large language models require expensive manual annotations. |
| Approach: | They propose an offline direct preference optimization method that collects preference pairs through iterative sampling and execution feedback to improve model confidence. |
| Outcome: | The proposed method improves performance on three reasoning tasks and shows a 3.6% improvement over the standard method. |
Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) is a widely used reinforcement learning from human feedback (RLHF) method across various domains. |
| Approach: | They propose an approach that automatically re-weights ambiguous content to reduce ambiguities by calculating semantic similarity from preference pairs. |
| Outcome: | The proposed approach outperforms state-of-the-art approaches in performance across multiple model scales and widely adopted benchmark datasets. |
SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for aligning Large Language Models with human values are limited and results of DPO are not resilient. |
| Approach: | They propose a self-guided direct preference optimization algorithm that incorporates a pilot term to steer the gradient flow during the optimization process. |
| Outcome: | The proposed method can generate human-preferred response up to 9.19% higher than previous methods. |
MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples (2025.coling-main)
Copied to clipboard
Shuo Xie, Fangzhi Zhu, Jiahui Wang, Lulu Wen, Wei Dai, Xiaowei Chen, Junxiong Zhu, Kai Zhou, Bo Zheng
| Challenge: | Existing preference optimization methods such as DPO and KTO are inherently derived from PPO, requiring a reference model that adds GPU memory resources and relies heavily on abundant preference data. |
| Approach: | They propose an algorithm that leverages the average likelihood of model responses to fit the reward function and maximizes the utilization of preference data. |
| Outcome: | The proposed algorithm outperforms DPO, ORPO, and SimPO on MT-Bench and Arena-Hard. |
Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for direct preference optimization assign equal importance to all tokens while humans focus on more meaningful parts. |
| Approach: | They propose to use a transport-based token weighting scheme to enhance direct preference optimization by emphasizing meaningful token pairs and de-emphasizing less relevant ones to yield a more contrastive reward difference estimate. |
| Outcome: | Extensive experiments have validated the proposed method in improving instruction-following ability across various settings. |