Aligning Large Language Models with Implicit Preferences from User-Generated Content

Challenge:	Existing preference learning methods rely heavily on curated data from humans or advanced LLMs, which is costly and difficult to scale.
Approach:	They propose a framework that leverages implicit preferences in unlabeled user-generated content to generate preference data.
Outcome:	The proposed framework transforms user-generated content into user queries and generates responses from the policy model.

Similar Papers

Overview

Challenge

Approach

Outcome

Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment (2025.coling-main)

Copied to clipboard

Jianfei Zhang, Jun Bai, Bei Li, Yanmeng Wang, Rumei Li, Chenghua Lin, Wenge Rong

Semantic Text Processing

Challenge:	Human values are inherently diverse, making it insufficient to align LLMs solely with general preferences.
Approach:	They propose a flexible paradigm for individual preference alignment that disentangles preference representation from text generation in LLMs.
Outcome:	The proposed method produces aligned quality and better than PEFT-based methods while reducing training time for each new individual preference by 80% to 90%.

Icon2: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation (2025.emnlp-main)

Copied to clipboard

Qiyuan Chen, Hongsen Huang, Qian Shao, Jiahe Chen, Jintai Chen, Hongxia Xu, Renjie Hua, Ren Chuan, Jian Wu

Semantic Text Processing

Challenge:	Large Language Models (LLMs) require high quality preference datasets to align with human preferences.
Approach:	They propose a framework that leverages inherent regulation of LLMs’ representation space for efficient and tailored preference dataset construction, named Icon2.
Outcome:	The proposed framework improves performance on benchmarks like AlpacaEval 2.0 and Arena-Hard while reducing computational costs by up to 48.1%.

GRAVITY: A Framework for Personalized Text Generation via Profile-Grounded Synthetic Preferences (2026.eacl-long)

Copied to clipboard

Priyanka Dey, Daniele Rosa, Wenqing Zheng, Daniel Barcklow, Jieyu Zhao, Emilio Ferrara

Text Generation

Challenge:	Personalization in LLMs often relies on costly human feedback or interaction logs, limiting scalability and neglecting deeper user attributes.
Approach:	They propose a framework for generating synthetic, profile-grounded preference data that captures users’ interests, values, beliefs, and personality traits.
Outcome:	The proposed framework improves on book descriptions for 400 Amazon users across multiple cultures, with user studies showing that outputs are preferred over 86% of the time.

From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment (2026.acl-long)

Copied to clipboard

Jia-Nan Li, Jian Guan, Songhao Wu, Wei Wu, Rui Yan

Robustness in NLP

Challenge:	Current approaches to align large language models assume uniform human preferences, overlooking the diversity inherent in human populations.
Approach:	They propose a framework for scalable personalized alignment of large language models . they establish a preference space characterizing psychological and behavioral dimensions .
Outcome:	The proposed framework improves on existing methods with an average of 17.06% accuracy gain across four benchmarks and a strong adaptation capability to novel preferences.

Learning Preference Model for LLMs via Automatic Preference Data Generation (2023.emnlp-main)

Copied to clipboard

Shijia Huang, Jianqiao Zhao, Yanyang Li, Liwei Wang

Challenge:	Existing training methods for large language models rely on human-annotated data.
Approach:	They propose to learn the preference model for LLMs via automatic preference data generation (AutoPM) using HHH-guided preference data, they show reliability and potential .
Outcome:	The proposed approach enables LLMs to learn human preferences and align with human values.

Aligning LLMs with Individual Preferences via Interaction (2025.coling-main)

Copied to clipboard

Shujin Wu, Yi R. Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, Heng Ji

Challenge:	Existing studies on LLMs alignment focus on generalizing their behavior to generalized values such as helpfulness, harmlessness, and honesty.
Approach:	They train large language models to "interact to align" to implicitly infer user preferences . they use a multi-turn preference dataset to generate a personalized alignment .
Outcome:	The proposed method enables dynamic, personalized alignment via interaction with a multi-turn preference dataset.

Beyond Online Sampling: Bridging Offline-to-Online Alignment via Dynamic Data Transformation for LLMs (2025.emnlp-main)

Copied to clipboard

Zhang Zhang, Guhao Feng, Jian Guan, Di He, Wei Wu

Responsible & Trustworthy NLP

Challenge:	Direct Preference Optimization (DPO) eliminates complex reward modeling in aligning large language models with human preferences, but its online variant faces significant efficiency bottlenecks due to costly real-time preference sampling and the reward model annotation.
Approach:	They propose a framework that transforms static datasets into dynamically adaptive equivalents without the need for an explicit reward model.
Outcome:	The proposed approach matches or exceeds the performance of a fully online DPO.

A Grounded Preference Model for LLM Alignment (2024.findings-acl)

Copied to clipboard

Tahira Naseem, Guangxuan Xu, Sarathkrishna Swaminathan, Asaf Yehudai, Subhajit Chaudhury, Radu Florian, Ramón Astudillo, Asim Munawar

Language Models

Challenge:	Large Language Models (LLMs) suffer from factual inconsistency and hallucination despite recent advances . training a preference model requires substantial human annotation, which is expensive and labor-intensive.
Approach:	They propose to generate synthetic grounded preference data and train a Grounded Preference Model to assess the overall quality of grounded responses.
Outcome:	The proposed model can generate much better grounded responses as judged by GPT4 and achieves the TRUE faithfulness Benchmark.

Mirroring Users: Towards Building Preference-aligned User Simulator with User Feedback in Recommendation (2026.acl-long)

Copied to clipboard

Tianjun Wei, Huizhong Guo, Yingpeng Du, Zhu Sun, Huang Chen, Dongxia Wang, Jie Zhang

Green & Sustainable NLP

Challenge:	Large Language Models lack specific task alignment and large-scale simulations are challenging due to their ambiguity, noise and massive volume.
Approach:	They propose a framework that leverages user feedback in RSs with advanced LLM capabilities to generate high-quality simulation data.
Outcome:	The proposed framework boosts the alignment with human preferences and in-domain reasoning capabilities of the fine-tuned LLMs.

Drift: Decoding-time Personalized Alignments with Implicit User Preferences (2025.findings-emnlp)

Copied to clipboard

Minbeom Kim, Kang-il Lee, Seongho Joo, Hwaran Lee, Thibaut Thonet, Kyomin Jung

Responsible & Trustworthy NLP

Challenge:	Drift personalizes large language models at decoding time with implicit user preferences . Unlike traditional Reinforcement Learning from Human Feedback, Drift operates in a training-free manner .
Approach:	They propose a framework that personalizes large language models at decoding time with implicit user preferences.
Outcome:	The proposed framework personalizes large language models at decoding time with implicit user preferences.