Rethinking Diverse Human Preference Learning through Principal Component Analysis (2025.findings-acl)
Copied to clipboard
| Challenge: | Decomposed Reward Models extract diverse human preferences from binary comparisons without fine-grained annotations. |
| Approach: | They propose a decomposed reward model that extracts diverse human preferences from binary comparisons without fine-grained annotations. |
| Outcome: | The proposed approach extracts diverse human preferences from binary comparisons without fine-grained annotations. |
Similar Papers
Not All Voices Are Rewarded Equally: Probing and Repairing Reward Models across Human Diversity (2025.findings-emnlp)
Copied to clipboard
Zihao Li, Feihao Fang, Xitong Zhang, Jiaru Zou, Zhining Liu, Wei Xiong, Ziwei Wu, Baoyu Jing, Jingrui He
| Challenge: | Using real-world datasets, we conduct the most comprehensive study to date, auditing various state-of-the-art reward models across nine sensitive attributes, including age, gender, ethnicity, etc. |
| Approach: | They propose a method to mitigate group disparities in reward modeling by using real-world data. |
| Outcome: | The proposed method is based on a population-based dataset with nine demographic attributes, including gender, ethnicity, age, gender, and ethnicity. |
From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment (2026.acl-long)
Copied to clipboard
| Challenge: | Current approaches to align large language models assume uniform human preferences, overlooking the diversity inherent in human populations. |
| Approach: | They propose a framework for scalable personalized alignment of large language models . they establish a preference space characterizing psychological and behavioral dimensions . |
| Outcome: | The proposed framework improves on existing methods with an average of 17.06% accuracy gain across four benchmarks and a strong adaptation capability to novel preferences. |
PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing approaches to align large language models with human preferences are limited by their large-scale annotation and prone to reward overoptimization. |
| Approach: | They propose a training paradigm that integrates three complementary strategies to address these challenges by reformulating question–answer pairs into preference-task instructions, averaging the rewards aggregated from diverse preference- task instructions for each sample, and a balancing outputs from the value head under different dropout rates. |
| Outcome: | Experiments on public datasets show that PIRA improves performance considerably, enhances generalization, and effectively mitigates reward overoptimization. |
MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing reward models assume a global reward function, limiting personalization and pluralistic alignment. |
| Approach: | They propose a framework that leverages binary preference datasets to enhance personalized preference learning. |
| Outcome: | The proposed framework captures diverse human preferences without fine-grained annotations and significantly improves personalized preference learning on downstream tasks. |
Dissecting Human and LLM Preferences (2024.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that human and Large Language Model preferences are important for model fine-tuning and evaluation. |
| Approach: | They dissect the preferences of human and 32 different Large Language Models to understand their quantitative composition. |
| Outcome: | The proposed model is compared with 32 different large language models using real-world user-model conversations. |
Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment (2025.coling-main)
Copied to clipboard
| Challenge: | Human values are inherently diverse, making it insufficient to align LLMs solely with general preferences. |
| Approach: | They propose a flexible paradigm for individual preference alignment that disentangles preference representation from text generation in LLMs. |
| Outcome: | The proposed method produces aligned quality and better than PEFT-based methods while reducing training time for each new individual preference by 80% to 90%. |
Comparison-based Active Preference Learning for Multi-dimensional Personalization (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models have shown remarkable success, but aligning them with human preferences remains a core challenge. |
| Approach: | They propose to capture implicit user preferences from comparative feedback to improve model performance. |
| Outcome: | The proposed framework is able to capture implicit user preferences from comparative feedback. |
When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Reinforcement Learning from Human Feedback assumes homogeneous preferences across users . personalization can introduce up to 20% safety misalignment . |
| Approach: | They propose a framework to assess personalized preference learning by tailoring preferences for users . they compare eight personalization methods across three preference datasets . |
| Outcome: | The proposed framework measures performance, fairness, unintended effects, adaptability across preferences . performance differences between personalization methods could reach 36% when users strongly disagree . |
Uncovering Factor-Level Preference to Improve Human-Model Alignment (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models exhibit tendencies that diverge from human preferences, such as favoring certain writing styles or producing overly verbose outputs. |
| Approach: | They propose a framework to uncover and measure factor-level preference alignment of humans and large language models (LLMs) |
| Outcome: | The proposed framework uncovers and measures factor-level preference alignment of humans and large language models. |
Personalized LLM Decoding via Contrasting Personal Preference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Personalization of large language models (LLMs) is becoming increasingly important as they are increasingly deployed in real-world applications. |
| Approach: | They propose a decoding-time approach that leverages the user's implicit reward signal by performing parameter-efficient fine-tuning on user-specific data. |
| Outcome: | The proposed approach improves personalization by an average of 10.57% in ROUGE-L without external reward models or additional training procedures. |