Evaluating Conversational Agents with Persona-driven User Simulations based on Large Language Models: A Sales Bot Case Study (2025.emnlp-industry)
Copied to clipboard
Justyna Gromada, Alicja Kasicka, Ewa Komkowska, Lukasz Krajewski, Natalia Krawczyk, Morgan Veyret, Bartosz Przybył, Lina M. Rojas-Barahona, Michał K. Szczerbak
| Challenge: | Recent advances in LLMs enable sophisticated user simulations that can replace traditional rule-based evaluations. |
| Approach: | They propose a persona-driven approach to conversational agent evaluation using Large Language Models (LLMs) they introduce a dataset of customer personas, which are then used to configure a single LLM-based user simulator. |
| Outcome: | The proposed model emulates nuanced customer roles and can implement cross-selling strategies with minimal impact on customer satisfaction, varying by customer type. |
Similar Papers
Characteristic AI Agents via Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Commercial products have been devoted to creating character-driven chatbots using large language models, but academic research in this area remains relatively scarce. |
| Approach: | They investigate the performance of LLMs in constructing characteristic AI agents by simulating real-life individuals across different settings. |
| Outcome: | The proposed benchmark compared LLMs with real-life individuals in different settings and includes evaluation metrics. |
PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have shown that LLMs can generate content that aligns with their assigned personality traits, but there is limited research on whether they consistently reflect specific personality traits. |
| Approach: | They propose to study the behavior of LLM-based agents which they refer to as LLM personas and simulate them to measure their personality traits. |
| Outcome: | The proposed model is based on the Big Five personality model and has been validated by human evaluations and automatic evaluations. |
A Survey on LLM-based Conversational User Simulation (2026.eacl-long)
Copied to clipboard
Bo Ni, Yu Wang, Leyao Wang, Branislav Kveton, Franck Dernoncourt, Yu Xia, Hongjie Chen, Reuben Luera, Samyadeep Basu, Subhojyoti Mukherjee, Puneet Mathur, Nesreen K. Ahmed, Junda Wu, Li Li, Huixin Zhang, Ruiyi Zhang, Tong Yu, Sungchul Kim, Jiuxiang Gu, Zhengzhong Tu, Alexa Siu, Zichao Wang, Seunghyun Yoon, Nedim Lipka, Namyong Park, Zihao Lin, Trung Bui, Yue Zhao, Tyler Derr, Ryan A. Rossi
| Challenge: | Recent advances in large language models (LLMs) have enabled high-fidelity generation of synthetic user conversation. |
| Approach: | They propose a taxonomy covering user granularity and simulation objectives . they analyze core techniques and evaluation methodologies to help them understand the latest developments . |
| Outcome: | The proposed model enables high-fidelity generation of synthetic user conversation. |
Injecting Salesperson’s Dialogue Strategies in Large Language Models with Chain-of-Thought Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent research in dialogue systems focuses on task-oriented (TOD) and open-domain (chit-chat) dialogues. |
| Approach: | They propose to use chit-chat to simulate task-oriented dialogues to train sales agents. |
| Outcome: | The proposed model improves coherence and reduces aggression, improving model learning for sales-customer interactions. |
PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing research focuses on character-level settings and static evaluation formats fail to capture the complexity of everyday social interactions. |
| Approach: | They propose a dynamic simulation framework for evaluating and improving persona-level role-playing in large language models (LLMs). |
| Outcome: | The proposed framework leverages user-generated social content to construct a nuanced persona bank and elicits multi-turn, context-rich interactions within simulated social environments. |
SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluation (2025.acl-industry)
Copied to clipboard
| Challenge: | Recommender systems are a key component of our day-to-day lives, but evaluation remains a challenge due to the gap between offline metrics and online behaviors. |
| Approach: | They propose a framework that enables users to build believable human proxies from historical data. |
| Outcome: | The proposed framework exhibits closer alignment with real humans than previous work, both at micro and macro levels. |
Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation protocols for large language models (LLMs) are inadequate for conversational recommender systems. |
| Approach: | They propose an evaluation approach based on LLMs that harnesses LLM-based user simulators to evaluate ChatGPT's performance. |
| Outcome: | The proposed evaluation approach can simulate various system-user interaction scenarios. |
Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models show promise in simulating human-like behavior, raising the question of their ability to represent a diverse population of users. |
| Approach: | They propose a protocol to evaluate the degree to which language models can accurately emulate human behavior in conversational recommendation systems. |
| Outcome: | The proposed protocol evaluates five tasks to reveal deviations of language models from human behavior and offers insights on how to reduce deviations with model selection and prompting strategies. |
A Practical Approach for Building Production-Grade Conversational Agents with Workflow Graphs (2025.acl-industry)
Copied to clipboard
Chiwan Park, Wonjun Jang, Daeryong Kim, Aelim Ahn, Kichang Yang, Woosung Hwang, Jihyeon Roh, Hyerin Park, Hyosun Wang, Min Seok Kim, Jihoon Kang
| Challenge: | Large Language Models (LLMs) have led to significant improvements in various service domains, including search, recommendation, and chatbot applications. |
| Approach: | They propose a framework for developing scalable, controllable, and reliable AI-driven agents that can be applied to real-world applications. |
| Outcome: | The proposed framework bridges the gap between academic research and real-world application, and enables scalable, controllable, and reliable AI-driven agents. |
PersonaGym: Evaluating Persona Agents and LLMs (2025.findings-emnlp)
Copied to clipboard
Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik R Narasimhan, Vishvak Murahari
| Challenge: | Persona agents are LLM agents conditioned to act according to an assigned persona . evaluating how faithfully these agents adhere to their personas remains a challenge . |
| Approach: | a new study evaluates persona agents' ability to act according to an assigned persona . a persona agent's person score is a human-aligned automatic metric that can be used to evaluate a model . |
| Outcome: | a new evaluation framework and a human-aligned automatic metric show that persona agents can perform better. |