Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization

Challenge:	Direct Preference Optimization (DPO) is a widely used reinforcement learning from human feedback (RLHF) method across various domains.
Approach:	They propose an approach that automatically re-weights ambiguous content to reduce ambiguities by calculating semantic similarity from preference pairs.
Outcome:	The proposed approach outperforms state-of-the-art approaches in performance across multiple model scales and widely adopted benchmark datasets.

Similar Papers

Overview

Challenge

Approach

Outcome

Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization (2025.acl-long)

Copied to clipboard

Meng Li, Guangda Huzhang, Haibo Zhang, Xiting Wang, Anxiang Zeng

Challenge:	Existing methods for direct preference optimization assign equal importance to all tokens while humans focus on more meaningful parts.
Approach:	They propose to use a transport-based token weighting scheme to enhance direct preference optimization by emphasizing meaningful token pairs and de-emphasizing less relevant ones to yield a more contrastive reward difference estimate.
Outcome:	Extensive experiments have validated the proposed method in improving instruction-following ability across various settings.

Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering (2026.findings-acl)

Copied to clipboard

Anas Mohamed, Azal Ahmad Khan, Xinran Wang, Ahmad Faraz Khan, Shuwen Ge, Saman Bahzad Khan, Ayaan Ahmad, Ali Anwar

Language Models

Challenge:	Direct Preference Optimization (DPO) is an off-policy alternative to RL for automatic prompt engineering, but its token-level regularization leaves semantic inconsistency unchecked as prompts that win higher preference scores can still drift away from the user’s intended meaning.
Approach:	They propose a variant of Direct Preference Optimization that preserves semantic consistency while maintaining its simplicity and efficiency.
Outcome:	The proposed model outperforms state-of-the-art prompt optimization baselines and several DPO variants on three standard text-to-image prompt-optimization benchmarks and three language models.

What Do LLMs Learn First? Asymmetric Learning Dynamics of Input Complexity and Output Ambiguity in Preference Alignment (2026.acl-long)

Copied to clipboard

Mengyang Li, Jingwen Wang, Pinlong Zhao

Challenge:	Existing methods treat all preference pairs uniformly during training.
Approach:	They propose a training framework that maintains separate, adaptive pacing schedules for each dimension.
Outcome:	The proposed training framework outperforms curriculum baselines by 2.1% and 0.21 points . it achieves 42.3% length-controlled win rate on AlpacaEval 2.0 and 7.66 on MT-Bench .

Direct Preference Optimization with an Offset (2024.findings-acl)

Copied to clipboard

Afra Amini, Tim Vieira, Ryan Cotterell

Language Models

Challenge:	Direct preference optimization (DPO) fine-tunes language models with human preferences . but not all preference pairs are equal; sometimes, the preferred response is only slightly better than the dispreferred one.
Approach:	They propose a generalization of direct preference optimization that does not treat every preference pair equally during fine-tuning.
Outcome:	The proposed algorithm outperforms DPO on tasks with limited preference pairs . it requires the difference between likelihood of preferred and dispreferred response to be greater than offset value .

SPO: Self Preference Optimization with Self Regularization (2025.findings-emnlp)

Copied to clipboard

Yuhao Sun, Yifan Zhang, Quandong Wang, Qinzhuo Wu, Wei Liu, Jian Luan

Machine Translation

Challenge:	Existing reference-free preference optimization methods exhibit higher training efficiency but are prone to overoptimization, leading to performance degradation.
Approach:	They propose a reference-free preference optimization method that replaces the logsigmoid loss function with a SiLU function to improve the model's performance.
Outcome:	The proposed method achieves 7% improvement over SimPO on AlpacaEval 2 and MT-Bench.

Expectation Preference Optimization: Reliable Preference Estimation for Improving the Reasoning Capability of Large Language Models (2025.emnlp-main)

Copied to clipboard

Zelin Li, Dawei Song

Language Models

Challenge:	Pairwise preference optimization is used to improve supervised fine-tuning performance of large language models.
Approach:	They propose an algorithm that takes pairs of sample groups instead of single samples for preference learning.
Outcome:	The proposed algorithm outperforms baseline methods on reasoning benchmarks.

RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models (2024.findings-naacl)

Copied to clipboard

Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, Prathap Ramachandra

Language Models

Challenge:	Reinforcement learning with human feedback (RLHF) is widely employed to align large language models with user intent.
Approach:	They propose to combine rejection sampling and direct preference optimization to improve alignment with user intent by identifying pairs of contrastive samples from human annotator and alternative LLMs.
Outcome:	The proposed method outperforms existing methods including RS, PPO, and DPO in a limited resource environment.

Adaptive Preference Optimization with Uncertainty-aware Utility Anchor (2025.findings-emnlp)

Copied to clipboard

Xiaobo Wang, Zixia Jia, Jiaqi Li, Qi Liu, Zilong Zheng

Green & Sustainable NLP

Challenge:	Offline preference optimization methods are efficient for large language models (LLMs) alignment.
Approach:	They propose an offline preference optimization framework that estimates uncertainties from preference data . the method enables training even in scenarios where the data is unpaired .
Outcome:	The proposed method enables training even in scenarios where the data is unpaired .

Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness (2024.findings-emnlp)

Copied to clipboard

Jian Li, Haojing Huang, Yujia Zhang, Pengfei Xu, Xi Chen, Rui Song, Lida Shi, Jingwen Wang, Hao Xu

Language Models

Challenge:	Recent studies have focused on replacing the reward model in Reinforcement Learning with Human Feedback (RLHF) methods for Large Language Models (LLMs).
Approach:	They propose a self-supervised preference optimization framework that replaces the reward model with a preference loss and alignment loss to improve LLMs' ability to understand human preferences.
Outcome:	The proposed framework can be integrated with existing preference optimization methods and significantly boost their performance.

FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings (2025.acl-short)

Copied to clipboard

Tong Liu, Xiao Yu, Wenxuan Zhou, Jindong Gu, Volker Tresp

Challenge:	Efficient preference optimization algorithms such as Direct Preference Optimization (DPO) have become a popular approach in aligning large language models with human preferences.
Approach:	They propose a preference optimization variant that instead down-weighs misranked preference pairs and prioritizes enhancing the model’s understanding of pairs that it can already rank correctly.
Outcome:	The proposed model outperforms DPO on benchmarks like Alpaca Eval 2.0 and Arena-Hard using mistral-base-7B and Llama-3-Instruct-8B with the introduced hyperparameter fixed.