Enhancing Alignment using Curriculum Learning & Ranked Preferences (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) is an effective technique that leverages pairwise preference data to align LLMs to human preferences. |
| Approach: | They propose to use pairwise preference data to create multiple preference pairs for a given prompt. |
| Outcome: | The proposed method outperforms standard DPO on MTbench, Vicuna bench, and WizardLM with a score of 7.43 on the test sets. |
Similar Papers
Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks (2025.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in math reasoning problemsolving, text generation, summarization, creative writing, among other tasks. |
| Approach: | They evaluate Direct Preference Optimization and its variants for aligning Large Language Models with human preferences. |
| Outcome: | The proposed alignment methods achieve near-optimal performance even with smaller subsets of training data. |
MWPO: Enhancing LLMs Performance through Multi-Weight Preference Strength and Length Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing offline alternatives to Reinforcement Learning from Human Feedback (RLHF) are available at https://github.com/AIR-hl/MWPO. |
| Approach: | They propose an offline method to optimize preference pairs based on implicit reward margins and response length margins by reweighting them using a geometric mixture. |
| Outcome: | The proposed method outperforms state-of-the-art methods on four different scales and reduces generation length by 9.4%. |
Causal Direct Preference Optimization for Language Model Alignment (2026.findings-eacl)
Copied to clipboard
| Challenge: | Empirical evaluations show that CDPO surpasses DPO-based baselines by achieving unbiased fine-tuning through causal reasoning. |
| Approach: | They propose a framework that incorporates causal inference principles to mitigate the influence of confounders and sharpen the signal of genuine human preferences. |
| Outcome: | The proposed framework preserves the tractability of direct optimization while enhancing robustness to spurious correlations and annotation biases. |
Direct Preference Optimization with an Offset (2024.findings-acl)
Copied to clipboard
| Challenge: | Direct preference optimization (DPO) fine-tunes language models with human preferences . but not all preference pairs are equal; sometimes, the preferred response is only slightly better than the dispreferred one. |
| Approach: | They propose a generalization of direct preference optimization that does not treat every preference pair equally during fine-tuning. |
| Outcome: | The proposed algorithm outperforms DPO on tasks with limited preference pairs . it requires the difference between likelihood of preferred and dispreferred response to be greater than offset value . |
RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Reinforcement learning with human feedback (RLHF) is widely employed to align large language models with user intent. |
| Approach: | They propose to combine rejection sampling and direct preference optimization to improve alignment with user intent by identifying pairs of contrastive samples from human annotator and alternative LLMs. |
| Outcome: | The proposed method outperforms existing methods including RS, PPO, and DPO in a limited resource environment. |
Teaching Your Models to Understand Code via Focal Preference Alignment (2025.emnlp-main)
Copied to clipboard
Jie Wu, Haoling Li, Xin Zhang, Xiao Liu, Yangyu Huang, Jianwen Luo, Yizhen Zhang, Zuchao Li, Ruihang Chu, Yujiu Yang, Scarlett Li
| Challenge: | Existing methods for supervised fine-tuning focus on unit test feedback to construct preference pairs. |
| Approach: | They propose a preference alignment framework that mimics human iterative debugging to refine Code LLMs. |
| Outcome: | Experiments show that Preference Learning improves on BigCodeBench and BigCodeBind tasks. |
FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings (2025.acl-short)
Copied to clipboard
| Challenge: | Efficient preference optimization algorithms such as Direct Preference Optimization (DPO) have become a popular approach in aligning large language models with human preferences. |
| Approach: | They propose a preference optimization variant that instead down-weighs misranked preference pairs and prioritizes enhancing the model’s understanding of pairs that it can already rank correctly. |
| Outcome: | The proposed model outperforms DPO on benchmarks like Alpaca Eval 2.0 and Arena-Hard using mistral-base-7B and Llama-3-Instruct-8B with the introduced hyperparameter fixed. |
sDPO: Don’t Use Your Data All at Once (2025.coling-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly requiring precision and accuracy in alignment tuning. |
| Approach: | They propose a stepwise DPO technique that partitions available preference datasets incrementally rather than utilizing entire dataset simultaneously. |
| Outcome: | The proposed technique improves the accuracy of reference models and the overall performance of the final model. |
2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision (2025.findings-naacl)
Copied to clipboard
Shilong Li, Yancheng He, Hui Huang, Xingyuan Bu, Jiaheng Liu, Hangyu Guo, Weixun Wang, Jihao Gu, Wenbo Su, Bo Zheng
| Challenge: | Existing methods that optimize for scalar scores or ranking reward ignore multi-dimensional nature of human preferences. |
| Approach: | They propose to extend the preference of Direct Preference Optimization to two dimensions: segments and aspects. |
| Outcome: | The proposed framework decomposes the overall objective into multi-segment and multi-aspect objectives. |
DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization (2025.findings-acl)
Copied to clipboard
Amitava Das, Suranjana Trivedy, Danush Khanna, Yaswanth Narsupalli, Basab Ghosh, Rajarshi Roy, Gurpreet Singh, Vinija Jain, Vasu Sharma, Aishwarya Naresh Reganti, Aman Chadha
| Challenge: | Direct Preference Optimization (DPO) is a cornerstone for preference alignment but is constrained by fixed divergence measures and limited feature transformations. |
| Approach: | They propose a new enhancement of Direct Preference Optimization that integrates kernel methods to overcome these challenges. |
| Outcome: | The proposed model improves divergence measures and features by using kernels . the proposed model achieves state-of-the-art generalization in factuality, safety, reasoning, and instruction following . |