MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing reward models assume a global reward function, limiting personalization and pluralistic alignment. |
| Approach: | They propose a framework that leverages binary preference datasets to enhance personalized preference learning. |
| Outcome: | The proposed framework captures diverse human preferences without fine-grained annotations and significantly improves personalized preference learning on downstream tasks. |
Similar Papers
APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that RLHF improves performance of Large Language Models . BT-based RMs struggle to distinguish between similar preference responses . |
| Approach: | They propose to enhance BT-based reward models by using an adaptive margin mechanism . they use semantic similarity and reward-predicted reward differences to adjust focus . |
| Outcome: | Experimental results show that the proposed method outperforms existing methods in both in-distribution and OOD settings. |
When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Reinforcement Learning from Human Feedback assumes homogeneous preferences across users . personalization can introduce up to 20% safety misalignment . |
| Approach: | They propose a framework to assess personalized preference learning by tailoring preferences for users . they compare eight personalization methods across three preference datasets . |
| Outcome: | The proposed framework measures performance, fairness, unintended effects, adaptability across preferences . performance differences between personalization methods could reach 36% when users strongly disagree . |
PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing approaches to align large language models with human preferences are limited by their large-scale annotation and prone to reward overoptimization. |
| Approach: | They propose a training paradigm that integrates three complementary strategies to address these challenges by reformulating question–answer pairs into preference-task instructions, averaging the rewards aggregated from diverse preference- task instructions for each sample, and a balancing outputs from the value head under different dropout rates. |
| Outcome: | Experiments on public datasets show that PIRA improves performance considerably, enhances generalization, and effectively mitigates reward overoptimization. |
From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment (2026.acl-long)
Copied to clipboard
| Challenge: | Current approaches to align large language models assume uniform human preferences, overlooking the diversity inherent in human populations. |
| Approach: | They propose a framework for scalable personalized alignment of large language models . they establish a preference space characterizing psychological and behavioral dimensions . |
| Outcome: | The proposed framework improves on existing methods with an average of 17.06% accuracy gain across four benchmarks and a strong adaptation capability to novel preferences. |
Self-Guided Alignment: Adaptive Preference Sensing for Multi-Objective Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to align LLMs with diverse human values rely on ground-truth scores . existing approaches implicitly approximate an average-user preference, thereby failing to capture heterogeneity of human values or accommodate conflicting user needs. |
| Approach: | They propose a framework that transforms passive reward dependency into an intrinsic adaptive sensing capability. |
| Outcome: | The proposed framework outperforms state-of-the-art models in multiple model scales and improves preference alignment. |
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Reinforcement learning from human feedback (RLHF) is the primary method for aligning large language models with human preferences. |
| Approach: | They propose to train an Absolute-Rating Multi-Objective Reward Model with multi-dimensional absolute-rating data. |
| Outcome: | The proposed model outperforms the LLM-as-a-judge method on RewardBench . it achieves state-of-the-art performance on the benchmark . |
CoPL: Collaborative Preference Learning for Personalizing LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for personalizing large language models struggle with flexibility and generalization. |
| Approach: | They propose a graph-based collaborative filtering framework that models user-response relationships to enhance preference estimation in sparse annotation settings. |
| Outcome: | The proposed framework outperforms existing reward models in TL;DR, UltraFeedback-P, and PersonalLLM datasets. |
Personalized LLM Decoding via Contrasting Personal Preference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Personalization of large language models (LLMs) is becoming increasingly important as they are increasingly deployed in real-world applications. |
| Approach: | They propose a decoding-time approach that leverages the user's implicit reward signal by performing parameter-efficient fine-tuning on user-specific data. |
| Outcome: | The proposed approach improves personalization by an average of 10.57% in ROUGE-L without external reward models or additional training procedures. |
Aligning LLMs with Individual Preferences via Interaction (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies on LLMs alignment focus on generalizing their behavior to generalized values such as helpfulness, harmlessness, and honesty. |
| Approach: | They train large language models to "interact to align" to implicitly infer user preferences . they use a multi-turn preference dataset to generate a personalized alignment . |
| Outcome: | The proposed method enables dynamic, personalized alignment via interaction with a multi-turn preference dataset. |
Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for aligning language models with human preferences rely on reward signals and additional annotated data, limiting their scalability and adaptability to diverse human values. |
| Approach: | They propose a discriminative paradigm that leverages the intrinsic preference judgment capabilities of the model to align language models with human preferences. |
| Outcome: | The proposed model is scalable and efficient, paving the way for more adaptive personalized alignment. |