BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to align Large Language Models with human preferences fail to maintain general knowledge and alignment when faced with personalized preferences. |
| Approach: | They propose a method that utilizes the initial responses of the reference model to mitigate forgetting while accommodating personalized alignment. |
| Outcome: | The proposed approach mitigates forgetting while accommodating personalized alignment while preserving global knowledge and general alignment. |
Similar Papers
InfoPO: On Mutual Information Maximization for Large Language Model Alignment (2025.naacl-long)
Copied to clipboard
Teng Xiao, Zhen Ge, Sujay Sanghavi, Tian Wang, Julian Katz-Samuels, Marc Versage, Qingjun Cui, Trishul Chilimbi
| Challenge: | Recent studies have shown that direct preference optimization and its variants can be useful for fine-tuning large language models with human preferences data. |
| Approach: | They propose a preference fine-tuning algorithm that effectively and efficiently aligns large language models using preference data. |
| Outcome: | Extensive experiments show that the proposed algorithm outperforms established baselines on reasoning tasks. |
A Survey on Personalized Alignment—The Missing Piece for Large Language Models in Real-World Applications (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their transition to real-world applications reveals a critical limitation: the inability to adapt to individual preferences while maintaining alignment with universal human values. |
| Approach: | They propose a framework that enables LLMs to adapt their behavior within ethical boundaries based on individual preferences. |
| Outcome: | The proposed framework analyzes implementation approaches and evaluates their effectiveness across various scenarios. |
K-order Ranking Preference Optimization for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing list-wise methods focus on optimizing list ranking consistency for LLMs to improve ranking abilities. |
| Approach: | They propose to extend the Plackett-Luce model to accommodate top-K ranking by extending the DPO’s Plact-Lucer model to dynamically determine appropriate K for different samples. |
| Outcome: | The proposed model can be extended to accommodate top-K ranking and improve training efficiency. |
ComPO: Community Preferences for Language Model Personalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for training language models with human feedback rely on subjective preferences that are assumed to account for an "average" user . however, annotating preferences is inherently subjective and results in generic models that generate outputs not preferred by many user groups. |
| Approach: | They propose a method to personalize preference optimization in LMs by contextualizing the probability distribution of model outputs with the preference provider. |
| Outcome: | The proposed method improves performance by focusing on group-level preferences rather than individual feedback. |
DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization (2025.findings-acl)
Copied to clipboard
Amitava Das, Suranjana Trivedy, Danush Khanna, Yaswanth Narsupalli, Basab Ghosh, Rajarshi Roy, Gurpreet Singh, Vinija Jain, Vasu Sharma, Aishwarya Naresh Reganti, Aman Chadha
| Challenge: | Direct Preference Optimization (DPO) is a cornerstone for preference alignment but is constrained by fixed divergence measures and limited feature transformations. |
| Approach: | They propose a new enhancement of Direct Preference Optimization that integrates kernel methods to overcome these challenges. |
| Outcome: | The proposed model improves divergence measures and features by using kernels . the proposed model achieves state-of-the-art generalization in factuality, safety, reasoning, and instruction following . |
From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment (2026.acl-long)
Copied to clipboard
| Challenge: | Current approaches to align large language models assume uniform human preferences, overlooking the diversity inherent in human populations. |
| Approach: | They propose a framework for scalable personalized alignment of large language models . they establish a preference space characterizing psychological and behavioral dimensions . |
| Outcome: | The proposed framework improves on existing methods with an average of 17.06% accuracy gain across four benchmarks and a strong adaptation capability to novel preferences. |
MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples (2025.coling-main)
Copied to clipboard
Shuo Xie, Fangzhi Zhu, Jiahui Wang, Lulu Wen, Wei Dai, Xiaowei Chen, Junxiong Zhu, Kai Zhou, Bo Zheng
| Challenge: | Existing preference optimization methods such as DPO and KTO are inherently derived from PPO, requiring a reference model that adds GPU memory resources and relies heavily on abundant preference data. |
| Approach: | They propose an algorithm that leverages the average likelihood of model responses to fit the reward function and maximizes the utilization of preference data. |
| Outcome: | The proposed algorithm outperforms DPO, ORPO, and SimPO on MT-Bench and Arena-Hard. |
Weights-Rotated Preference Optimization for Large Language Models (2025.emnlp-main)
Copied to clipboard
Chenxu Yang, Ruipeng Jia, Mingyu Zheng, Naibin Gu, Zheng Lin, Siyuan Chen, Weichong Yin, Hua Wu, Weiping Wang
| Challenge: | Existing methods to align large language models with high reward hacking are limited by the complexity of the parameter space and the complexity. |
| Approach: | They propose a weights-rotated preference optimization algorithm that constrains the output layer logits with the KL divergence inherited from DPO and fine-tunes the intermediate hidden states. |
| Outcome: | The proposed algorithm achieves a 3.27-point improvement on AlpacaEval 2 and surpasses the best baseline by 6.2 to 7.5 points on MT-Bench with merely 0.015% of the trainable parameters. |
Adaptive Preference Optimization with Uncertainty-aware Utility Anchor (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Offline preference optimization methods are efficient for large language models (LLMs) alignment. |
| Approach: | They propose an offline preference optimization framework that estimates uncertainties from preference data . the method enables training even in scenarios where the data is unpaired . |
| Outcome: | The proposed method enables training even in scenarios where the data is unpaired . |
SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for preference optimization of large language models use pairs of positive and negative samples, but the quality of positive samples may become similar during training, complicating preference learning. |
| Approach: | SeaPO introduces error types commonly occurring in large language models to improve preference learning. |
| Outcome: | SeaPO introduces error types into model Preference Optimization to improve model performance . negative samples are more erroneous than positive samples, and preference-based training mitigates errors . |