Debiasing Online Preference Learning via Preference Feature Preservation (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent preference learning frameworks for large language models (LLMs) simplify human preferences with binary pairwise comparisons and scalar rewards. |
| Approach: | They propose a preference feature preservation framework that preserves the distribution of human preference features and maps them throughout the online preference learning process. |
| Outcome: | The proposed framework maintains the distribution of human preference features and utilizing such rich signals throughout the online preference learning process. |
Similar Papers
Learning Preference Model for LLMs via Automatic Preference Data Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing training methods for large language models rely on human-annotated data. |
| Approach: | They propose to learn the preference model for LLMs via automatic preference data generation (AutoPM) using HHH-guided preference data, they show reliability and potential . |
| Outcome: | The proposed approach enables LLMs to learn human preferences and align with human values. |
BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing offline DAP methods for aligning large language models with human preference are computationally expensive due to their two-stage training pipeline that consists of a reward modeling phase. |
| Approach: | They propose to align large language models to human desiderata from offline preference datasets by using an online approach. |
| Outcome: | The proposed approach improves performance across a wide range of tasks when training with the same amount of preference data. |
FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Data (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on personalizing conversational assistants to meet specific user preferences. |
| Approach: | They propose to use a dataset to analyze a problem where only a small set of preference annotations can be collected per user. |
| Outcome: | The proposed approach leverages high-level features discovered from the data, achieving the best overall performance. |
CoPL: Collaborative Preference Learning for Personalizing LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for personalizing large language models struggle with flexibility and generalization. |
| Approach: | They propose a graph-based collaborative filtering framework that models user-response relationships to enhance preference estimation in sparse annotation settings. |
| Outcome: | The proposed framework outperforms existing reward models in TL;DR, UltraFeedback-P, and PersonalLLM datasets. |
Beyond Online Sampling: Bridging Offline-to-Online Alignment via Dynamic Data Transformation for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) eliminates complex reward modeling in aligning large language models with human preferences, but its online variant faces significant efficiency bottlenecks due to costly real-time preference sampling and the reward model annotation. |
| Approach: | They propose a framework that transforms static datasets into dynamically adaptive equivalents without the need for an explicit reward model. |
| Outcome: | The proposed approach matches or exceeds the performance of a fully online DPO. |
Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown remarkable capabilities to understand and generate human languages, supporting applications such as question answering, coding, and psychological counseling. |
| Approach: | They propose strategies to save annotation budgets while achieving competitive or even better performances for iterative preference learning. |
| Outcome: | The proposed methods save annotation budgets while achieving better performance. |
MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples (2025.coling-main)
Copied to clipboard
Shuo Xie, Fangzhi Zhu, Jiahui Wang, Lulu Wen, Wei Dai, Xiaowei Chen, Junxiong Zhu, Kai Zhou, Bo Zheng
| Challenge: | Existing preference optimization methods such as DPO and KTO are inherently derived from PPO, requiring a reference model that adds GPU memory resources and relies heavily on abundant preference data. |
| Approach: | They propose an algorithm that leverages the average likelihood of model responses to fit the reward function and maximizes the utilization of preference data. |
| Outcome: | The proposed algorithm outperforms DPO, ORPO, and SimPO on MT-Bench and Arena-Hard. |
BiasDPO: Mitigating Bias in Language Models through Direct Preference Optimization (2024.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been shown to be effective in complex language tasks, but their potential to perpetuate biases poses significant concerns. |
| Approach: | They propose a new framework employing Direct Preference Optimization to mitigate biases in LLMs. |
| Outcome: | The proposed model outperforms the baseline model on almost all bias benchmarks and achieves better performance than open-source models. |
Learning LLM Preference over Intra-Dialogue Pairs: A Framework for Utterance-level Understandings (2025.naacl-industry)
Copied to clipboard
Xuanqing Liu, Luyang Kong, Wei Niu, Afshin Khashei, Belinda Zeng, Steve Johnson, Jon Jay, Davor Golac, Matt Pope
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities in handling complex dialogue tasks without requiring use case-specific fine-tuning. |
| Approach: | They propose a framework that combines the scalability of LLM-generated labels with the precision of human annotations to achieve higher speed and accuracy comparable to larger models. |
| Outcome: | The proposed framework significantly improves accuracy across utterance-level dialogue tasks, including sentiment detection (over 2%), dialogue act classification (over 1.5%), etc. |
User Feedback Alignment for LLM-powered Exploration in Large-scale Recommendation Systems (2025.acl-industry)
Copied to clipboard
Jianling Wang, Yifan Liu, Yinghao Sun, Xuejian Ma, Yueqi Wang, He Ma, Zhengyang Su, Minmin Chen, Mingyan Gao, Onkar Dalal, Ed H. Chi, Lichan Hong, Ningren Han, Haokai Lu
| Challenge: | Large Language Models (LLMs) can be used to broaden user experiences beyond established preferences and reinforce feedback loops. |
| Approach: | They propose a hierarchical approach that combines hierarchic planning with LLM inference-time scaling to improve recommendation relevancy without compromising novelty. |
| Outcome: | The proposed approach shows significant gains in both user satisfaction and exploration diversity. |