HORIZON: A Benchmark for In-the-wild User Behaviour Modeling

Challenge:	Existing user modeling benchmarks focus on short sessions and next-item prediction within a single domain.
Approach:	They propose a benchmark that reformulates user modeling along three axes . it covers 54M users and 35M items, enabling pretraining and evaluation . they propose tasks and evaluation setups that better reflect real-world deployment scenarios .
Outcome:	The proposed benchmark covers 54M users and 35M items, and is based on Amazon Reviews.

Similar Papers

Overview

Challenge

Approach

Outcome

PersonaX: A Recommendation Agent-Oriented User Modeling Framework for Long Behavior Sequence (2025.findings-acl)

Copied to clipboard

Yunxiao Shi, Wujiang Xu, Zhang Zeqi, Xing Zi, Qiang Wu, Min Xu

Challenge:	Existing methods for user profile modeling extract only partial segments from full historical behavior sequence, resulting in incomplete modeling and suboptimal profiling.
Approach:	They propose an agent-agnostic LLM-UM framework to augment downstream recommendation agents . it segments complete historical behaviors into clustered groups and performs offline multi-persona profiling .
Outcome:	The proposed framework improves agent performance and inference efficiency by 31% and 10% using 30–50% of behavioral data.

AlignUSER: Human-Aligned LLM Agents via World Models for Recommender System Evaluation (2026.acl-long)

Copied to clipboard

Nicolas Bougie, Gian Maria Marconi, Xiaotong Ye, Narimawa Watanabe

Language Models

Challenge:	Existing evaluation practices for recommender systems rely on few-shot prompting and offline metrics are often misaligned with online behavior.
Approach:	They propose a framework that learns world-model-driven agents from human interactions.
Outcome:	The proposed framework enables agents to express rich preferences and feedback in natural language and interact with recommender systems in a simulation.

DEEPER Insight into Your User: Directed Persona Refinement for Dynamic Persona Modeling (2025.acl-long)

Copied to clipboard

Aili Chen, Chengyu Du, Jiangjie Chen, Jinghan Xu, Yikai Zhang, Siyu Yuan, Zulong Chen, Liangyue Li, Yanghua Xiao

Challenge:	Existing methods for generating personas from static historical data fail to capture dynamic behaviors and evolving preferences in real-world interactive scenarios.
Approach:	They propose a novel approach that iteratively updates personas using streaming user behavior data to continually enhance their quality.
Outcome:	The proposed approach delivers 32.2% reduction in user behavior prediction error over four update rounds, outperforming the best baseline by 22.92%.

User Behavior Prediction as a Generic, Robust, Scalable, and Low-Cost Evaluation Strategy for Estimating Generalization in LLMs (2025.findings-acl)

Copied to clipboard

Sougata Saha, Monojit Choudhury

Language Models

Challenge:	We argue that knowledge-retrieval and reasoning tasks are not ideal for measuring generalization, as LLMs are not trained for specific tasks.
Approach:	They propose a statistically motivated framework using personalization to assess generalization in Large Language Models.
Outcome:	The proposed framework outperforms existing models on movie and music recommendation datasets, but all models have room for improvement, especially Llama.

A User-Centric Multi-Intent Benchmark for Evaluating Large Language Models (2024.emnlp-main)

Copied to clipboard

Jiayin Wang, Fengran Mo, Weizhi Ma, Peijie Sun, Min Zhang, Jian-Yun Nie

Semantic Text Processing

Challenge:	Existing benchmarks focus on specific predefined model abilities, such as world knowledge, reasoning, etc., making it difficult for users to determine which LLM best suits their particular needs.
Approach:	They propose to evaluate large language models from a user-centric perspective and use real-world use cases to identify their effectiveness under distinct intents.
Outcome:	The proposed benchmarks achieve a correlation between human preference and the user-reported scenarios and human intents.

LifeSim: Long-Horizon User Life Simulator for Personalized Assistant Evaluation (2026.findings-acl)

Copied to clipboard

Feiyu Duan, Xuanjing Huang, Zhongyu Wei

Challenge:	Existing benchmarks for personalized assistants fail to capture the complexity of external contexts and users’ cognitive states.
Approach:	They propose a user simulator that models user cognition through the Belief-Desire-Intention model within physical environments for coherent life trajectories generation and simulates intention-driven user interactive behaviors.
Outcome:	The proposed model can model user cognition through the Belief-Desire-Intention model within physical environments for coherent life trajectories generation and simulates intention-driven user interactive behaviors.

GuideBench: Benchmarking Domain-Oriented Guideline Following for LLM Agents (2025.acl-long)

Copied to clipboard

Lingxiao Diao, Xinyue Xu, Wanxuan Sun, Cheng Yang, Zhuosheng Zhang

Commonsense Reasoning

Challenge:	Large language models (LLMs) have been widely deployed as autonomous agents capable of following user instructions and making decisions in real-world applications.
Approach:	They propose a benchmark to evaluate LLMs' ability to follow domain-oriented guidelines . they evaluate Lms on three critical aspects: adherence to diverse rules, robustness to rule updates .
Outcome:	The proposed benchmark evaluates LLMs on three critical aspects: adherence to diverse rules, robustness to rule updates, and alignment with human preferences.

SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluation (2025.acl-industry)

Copied to clipboard

Nicolas Bougie, Narimawa Watanabe

Language Models

Challenge:	Recommender systems are a key component of our day-to-day lives, but evaluation remains a challenge due to the gap between offline metrics and online behaviors.
Approach:	They propose a framework that enables users to build believable human proxies from historical data.
Outcome:	The proposed framework exhibits closer alignment with real humans than previous work, both at micro and macro levels.

LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces (2026.findings-acl)

Copied to clipboard

Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Shenglin Zhang, Kaipeng Zhang

Multimodality

Challenge:	Existing benchmarks for agentic programming in long-horizon command-line interface tasks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics.
Approach:	They propose a benchmark to evaluate agentic capabilities across long-horizon command-line interface tasks.
Outcome:	The proposed benchmarks cover four engineering categories: from scratch, feature addition, bug fixing, and refactoring.

Beyond Itinerary Planning—A Real-World Benchmark for Multi-Turn and Tool-Using Travel Tasks (2026.acl-long)

Copied to clipboard

Xiang Cheng, Yulan Hu, Xiangwen Zhang, Lu Xu, Lide Tan, Zheng Pan, Xin Li, Yong Liu

Responsible & Trustworthy NLP

Challenge:	Existing studies on LLM performance on travel planning have shown that existing settings are limited due to limited domain coverage, insufficient modeling of users’ implicit preferences in multi-turn conversations, and a lack of evaluation of agents’ capability boundaries.
Approach:	They propose a benchmark to evaluate LLMs' planning and tool-use abilities in real-world settings by collecting user queries, user preferences, and tools from real scenarios.
Outcome:	The proposed benchmark evaluates agents' capabilities in real-world settings and shows that even advanced models exhibit imbalanced performance across different capabilities.