Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models generate unintended outputs due to their unsupervised nature. |
| Approach: | They propose a method to construct preference pairs of selected and rejected LLMs by repeated random sampling to improve alignment performance. |
| Outcome: | The proposed method improves performance as the sample size increases. |
Similar Papers
RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Reinforcement learning with human feedback (RLHF) is widely employed to align large language models with user intent. |
| Approach: | They propose to combine rejection sampling and direct preference optimization to improve alignment with user intent by identifying pairs of contrastive samples from human annotator and alternative LLMs. |
| Outcome: | The proposed method outperforms existing methods including RS, PPO, and DPO in a limited resource environment. |
Beyond Online Sampling: Bridging Offline-to-Online Alignment via Dynamic Data Transformation for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) eliminates complex reward modeling in aligning large language models with human preferences, but its online variant faces significant efficiency bottlenecks due to costly real-time preference sampling and the reward model annotation. |
| Approach: | They propose a framework that transforms static datasets into dynamically adaptive equivalents without the need for an explicit reward model. |
| Outcome: | The proposed approach matches or exceeds the performance of a fully online DPO. |
Uncertainty-Aware Iterative Preference Optimization for Enhanced LLM Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for enhancing the performance of large language models require expensive manual annotations. |
| Approach: | They propose an offline direct preference optimization method that collects preference pairs through iterative sampling and execution feedback to improve model confidence. |
| Outcome: | The proposed method improves performance on three reasoning tasks and shows a 3.6% improvement over the standard method. |
Robust Preference Optimization via Dynamic Target Margins (2025.findings-acl)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) is an efficient method for ensuring safety and reliability in practical applications. |
| Approach: | They propose a dynamic target margin preference optimization algorithm that adjusts reward margins at the pairwise level. |
| Outcome: | The proposed method achieves an average 4.4% improvement over baselines, setting new benchmarks for state-of-the-art performance. |
Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling (2025.acl-long)
Copied to clipboard
| Challenge: | Existing safety alignment methods for Large Language Models (LLMs) face the distribution shift issue, which requires significant computational resources. |
| Approach: | They propose a framework that leverages the model’s intrinsic safety judgment capability to extract reward signals, which are then used to calculate label confidence for preference reordering. |
| Outcome: | The proposed framework improves safety performance while avoiding 300x computational overheads. |
MWPO: Enhancing LLMs Performance through Multi-Weight Preference Strength and Length Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing offline alternatives to Reinforcement Learning from Human Feedback (RLHF) are available at https://github.com/AIR-hl/MWPO. |
| Approach: | They propose an offline method to optimize preference pairs based on implicit reward margins and response length margins by reweighting them using a geometric mixture. |
| Outcome: | The proposed method outperforms state-of-the-art methods on four different scales and reduces generation length by 9.4%. |
Enhancing Alignment using Curriculum Learning & Ranked Preferences (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) is an effective technique that leverages pairwise preference data to align LLMs to human preferences. |
| Approach: | They propose to use pairwise preference data to create multiple preference pairs for a given prompt. |
| Outcome: | The proposed method outperforms standard DPO on MTbench, Vicuna bench, and WizardLM with a score of 7.43 on the test sets. |
MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples (2025.coling-main)
Copied to clipboard
Shuo Xie, Fangzhi Zhu, Jiahui Wang, Lulu Wen, Wei Dai, Xiaowei Chen, Junxiong Zhu, Kai Zhou, Bo Zheng
| Challenge: | Existing preference optimization methods such as DPO and KTO are inherently derived from PPO, requiring a reference model that adds GPU memory resources and relies heavily on abundant preference data. |
| Approach: | They propose an algorithm that leverages the average likelihood of model responses to fit the reward function and maximizes the utilization of preference data. |
| Outcome: | The proposed algorithm outperforms DPO, ORPO, and SimPO on MT-Bench and Arena-Hard. |
SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for preference optimization of large language models use pairs of positive and negative samples, but the quality of positive samples may become similar during training, complicating preference learning. |
| Approach: | SeaPO introduces error types commonly occurring in large language models to improve preference learning. |
| Outcome: | SeaPO introduces error types into model Preference Optimization to improve model performance . negative samples are more erroneous than positive samples, and preference-based training mitigates errors . |
Weights-Rotated Preference Optimization for Large Language Models (2025.emnlp-main)
Copied to clipboard
Chenxu Yang, Ruipeng Jia, Mingyu Zheng, Naibin Gu, Zheng Lin, Siyuan Chen, Weichong Yin, Hua Wu, Weiping Wang
| Challenge: | Existing methods to align large language models with high reward hacking are limited by the complexity of the parameter space and the complexity. |
| Approach: | They propose a weights-rotated preference optimization algorithm that constrains the output layer logits with the KL divergence inherited from DPO and fine-tunes the intermediate hidden states. |
| Outcome: | The proposed algorithm achieves a 3.27-point improvement on AlpacaEval 2 and surpasses the best baseline by 6.2 to 7.5 points on MT-Bench with merely 0.015% of the trainable parameters. |