| Challenge: | Presently, mainstream approaches to HPA heavily depend on fine-tuning . however, the huge computational and annotation costs of fine-timing are hard to ignore . |
| Approach: | They propose a tuning-free approach to HPA using LLMs' decoding . they first rethink the derivation procedures of DPO and build an instant scorer . |
| Outcome: | The proposed approach outperforms existing methods even with tuning-free baselines and an upgraded scorer. |
Similar Papers
SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for aligning Large Language Models with human values are limited and results of DPO are not resilient. |
| Approach: | They propose a self-guided direct preference optimization algorithm that incorporates a pilot term to steer the gradient flow during the optimization process. |
| Outcome: | The proposed method can generate human-preferred response up to 9.19% higher than previous methods. |
Direct Preference Optimization with an Offset (2024.findings-acl)
Copied to clipboard
| Challenge: | Direct preference optimization (DPO) fine-tunes language models with human preferences . but not all preference pairs are equal; sometimes, the preferred response is only slightly better than the dispreferred one. |
| Approach: | They propose a generalization of direct preference optimization that does not treat every preference pair equally during fine-tuning. |
| Outcome: | The proposed algorithm outperforms DPO on tasks with limited preference pairs . it requires the difference between likelihood of preferred and dispreferred response to be greater than offset value . |
InfoPO: On Mutual Information Maximization for Large Language Model Alignment (2025.naacl-long)
Copied to clipboard
Teng Xiao, Zhen Ge, Sujay Sanghavi, Tian Wang, Julian Katz-Samuels, Marc Versage, Qingjun Cui, Trishul Chilimbi
| Challenge: | Recent studies have shown that direct preference optimization and its variants can be useful for fine-tuning large language models with human preferences data. |
| Approach: | They propose a preference fine-tuning algorithm that effectively and efficiently aligns large language models using preference data. |
| Outcome: | Extensive experiments show that the proposed algorithm outperforms established baselines on reasoning tasks. |
sDPO: Don’t Use Your Data All at Once (2025.coling-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly requiring precision and accuracy in alignment tuning. |
| Approach: | They propose a stepwise DPO technique that partitions available preference datasets incrementally rather than utilizing entire dataset simultaneously. |
| Outcome: | The proposed technique improves the accuracy of reference models and the overall performance of the final model. |
Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to align English LLMs with human preferences rely on expensive human annotations or advanced multilingual preference alignment models. |
| Approach: | They propose a method that captures learned preferences from English models by implicit rewards . they annotate preference relations in cross-lingual instruction-following pairs using English . |
| Outcome: | The proposed approach captures learned preferences from well-aligned English models by implicit rewards and transfers them to other languages through iterative training. |
Beyond Online Sampling: Bridging Offline-to-Online Alignment via Dynamic Data Transformation for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) eliminates complex reward modeling in aligning large language models with human preferences, but its online variant faces significant efficiency bottlenecks due to costly real-time preference sampling and the reward model annotation. |
| Approach: | They propose a framework that transforms static datasets into dynamically adaptive equivalents without the need for an explicit reward model. |
| Outcome: | The proposed approach matches or exceeds the performance of a fully online DPO. |
Causal Direct Preference Optimization for Language Model Alignment (2026.findings-eacl)
Copied to clipboard
| Challenge: | Empirical evaluations show that CDPO surpasses DPO-based baselines by achieving unbiased fine-tuning through causal reasoning. |
| Approach: | They propose a framework that incorporates causal inference principles to mitigate the influence of confounders and sharpen the signal of genuine human preferences. |
| Outcome: | The proposed framework preserves the tractability of direct optimization while enhancing robustness to spurious correlations and annotation biases. |
RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Reinforcement learning with human feedback (RLHF) is widely employed to align large language models with user intent. |
| Approach: | They propose to combine rejection sampling and direct preference optimization to improve alignment with user intent by identifying pairs of contrastive samples from human annotator and alternative LLMs. |
| Outcome: | The proposed method outperforms existing methods including RS, PPO, and DPO in a limited resource environment. |
Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Empirical evaluations on eight recent LLMs reveal that DRPO significantly enhances alignment performance, enabling base models to outperform their SFT/RLHF-tuned counterparts. |
| Approach: | They propose a tuning-free approach to self-alignment called Dynamic Rewarding with Prompt Optimization (DRPO) it leverages a dynamic rewarding mechanism to identify and rectify alignment weaknesses . |
| Outcome: | The proposed approach outperforms existing methods and is highly adaptable to various alignment challenges. |
Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks (2025.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in math reasoning problemsolving, text generation, summarization, creative writing, among other tasks. |
| Approach: | They evaluate Direct Preference Optimization and its variants for aligning Large Language Models with human preferences. |
| Outcome: | The proposed alignment methods achieve near-optimal performance even with smaller subsets of training data. |