InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews (2024.acl-long)
Copied to clipboard
Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, Yanghua Xiao
| Challenge: | Existing methods focus on knowledge and linguistic patterns of characters. |
| Approach: | They propose to evaluate character fidelity of role-playing agents with psychological scales . they propose to use psychological scale to measure personality traits of RPAs based on personality traits. |
| Outcome: | The proposed model reproduces character fidelity with psychological scales and shows that it is effective in measuring personality traits. |
Similar Papers
EmoCharacter: Evaluating the Emotional Fidelity of Role-Playing Agents in Dialogues (2025.naacl-long)
Copied to clipboard
| Challenge: | EmoCharacter evaluates emotional fidelity of role-playing agents in dialogues . current evaluations focus on personality fidelity, tone imitation, and knowledge consistency . |
| Approach: | They propose a benchmark to assess emotional fidelity of role-playing agents in dialogues using large language models. |
| Outcome: | The proposed benchmark measures emotional fidelity of role-playing agents and the characters they portray. |
Capturing Minds, Not Just Words: Enhancing Role-Playing Language Models with Personality-Indicative Data (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing role-playing models focus on character knowledge and tones, but lack personality-indicative data to capture characters' minds. |
| Approach: | They propose to enhance role-playing agents (RPAs) via personality-indicative data by asking psychological scales to capture broad aspects of personality traits in individuals. |
| Outcome: | The proposed model exhibits advanced role-playing capabilities for both general and personality-related evaluations. |
Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents (2025.findings-acl)
Copied to clipboard
| Challenge: | Role-Playing Agents (RPAs) are increasingly popular due to diverse task requirements and agent designs. |
| Approach: | They propose an evidence-based evaluation design guideline for LLM-based RPAs based on agent attributes, task attributes, and evaluation metrics. |
| Outcome: | The proposed evaluation design guideline is based on a systematic review of 1,676 papers published between Jan. 2021 and Dec. 2024. |
Investigating the Personality Consistency in Quantized Role-Playing Dialogue Agents (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Using the Big Five personality traits model, we evaluate how stable assigned personalities are for Quantized Role-Playing Dialog Agents (QRPDA) during multi-turn interactions. |
| Approach: | They propose a non-parametric method to evaluate the stability of assigned personalities in quantized large language models (LLMs) for role-playing scenarios. |
| Outcome: | The proposed method shows that it maintains consistent personality traits in QRPDA, and it is more reliable in real-world applications. |
CharacterCraft: Bridging the Literature-Reality Dialogue Gap for Practical Role-Playing Agents (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing dialogue datasets have a bias between query distributions and real-world user language usage. |
| Approach: | They propose a framework for Chinese role-playing and a robust evaluation method . they propose specialized Chinese dialogue extraction model and specialized memory retrieval module . |
| Outcome: | The proposed framework extracts character dialogue from novels and ensures high data quality. |
PersonaGym: Evaluating Persona Agents and LLMs (2025.findings-emnlp)
Copied to clipboard
Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik R Narasimhan, Vishvak Murahari
| Challenge: | Persona agents are LLM agents conditioned to act according to an assigned persona . evaluating how faithfully these agents adhere to their personas remains a challenge . |
| Approach: | a new study evaluates persona agents' ability to act according to an assigned persona . a persona agent's person score is a human-aligned automatic metric that can be used to evaluate a model . |
| Outcome: | a new evaluation framework and a human-aligned automatic metric show that persona agents can perform better. |
Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have catalyzed numerous AI applications, among which role-playing agents (RPAs) are particularly popular. |
| Approach: | They propose to evaluate LLMs' character understanding capability via the character profiling task, i.e., summarizing character profiles from corresponding materials, a widely adopted yet understudied practice for RPA development. |
| Outcome: | The proposed model outperforms existing models and literature summarization methods and proves its ability to understand fictional characters in downstream tasks. |
ThinkPersona: Thinking with Persona Graphs for Faithful Individualized Role-Playing (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models are increasingly utilized as role-playing agents to simulate personas in interactive settings. |
| Approach: | They propose a role-playing agent trained to explicitly ground responses in individual identity. |
| Outcome: | The proposed agent can generate persona-consistent responses in long-context dialogues while maintaining general instruction-following capabilities. |
Modeling, Evaluating, and Embodying Personality in LLMs: A Survey (2025.findings-emnlp)
Copied to clipboard
Iago Alves Brito, Julia Soares Dollis, Fernanda Bufon Färber, Pedro Schindler Freire Brasil Ribeiro, Rafael Teixeira Sousa, Arlindo Rodrigues Galvão Filho
| Challenge: | This survey provides a comprehensive overview of the LLM-driven personality scenario. |
| Approach: | This survey provides a comprehensive overview of the LLM-driven personality scenario. |
| Outcome: | The proposed taxonomy analyzes the limitations of existing methods and identifies key research gaps. |
CharacterGPT: A Persona Reconstruction Framework for Role-Playing Agents (2025.naacl-industry)
Copied to clipboard
| Challenge: | Maintaining consistent character personas remains a significant challenge due to variability in information extraction. |
| Approach: | They propose a framework to dynamically reconstruct character personas through Character Persona Training. |
| Outcome: | The proposed framework is evaluated through Big Five personality evaluations and creative tasks, in which characters generate original narratives. |