CharacterBox: Evaluating the Role-Playing Capabilities of LLMs in Text-Based Virtual Worlds

Challenge:	Evaluating role-playing capabilities in large language models is challenging due to complex dynamics involved in role-playering.
Approach:	They propose a simulation sandbox that generates situational fine-grained character behavior trajectories to enhance LLM performance.
Outcome:	The proposed model generates situational fine-grained character behavior trajectories to enhance performance.

Similar Papers

Overview

Challenge

Approach

Outcome

Crafting Customisable Characters with LLMs: A Persona-Driven Role-Playing Agent Framework (2025.findings-emnlp)

Copied to clipboard

Bohao Yang, Dong Liu, Chenghao Xiao, Kun Zhao, Chen Tang, Chao Li, Lin Yuan, Yang Guang, Chenghua Lin

Challenge:	Large Language Models (LLMs) are capable of generating human-like text, but the potential for freely customisable characters remains underexplored.
Approach:	They propose a framework which employs Large Language Models to create freely customisable characters through personalised characteristic feature injection.
Outcome:	The proposed framework provides valuable insights for developing more accurate and customisable human simulacra.

PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models (2026.findings-acl)

Copied to clipboard

Wenlong Shi, Jianxun Lian, Mingqi Wu, Haiming Qin, Mingyang Zhou, Xing Xie, Naipeng Chao, Hao Liao

Semantic Text Processing

Challenge:	Existing research focuses on character-level settings and static evaluation formats fail to capture the complexity of everyday social interactions.
Approach:	They propose a dynamic simulation framework for evaluating and improving persona-level role-playing in large language models (LLMs).
Outcome:	The proposed framework leverages user-generated social content to construct a nuanced persona bank and elicits multi-turn, context-rich interactions within simulated social environments.

NarrativePlay: Interactive Narrative Understanding (2024.eacl-demo)

Copied to clipboard

Runcong Zhao, Wenjia Zhang, Jiazheng Li, Lixing Zhu, Yanran Li, Yulan He, Lin Gui

Challenge:	Existing systems for interactive agents focus on specific capabilities in predetermined scenarios.
Approach:	They propose a novel system that allows users to role-play a fictional character and interact with other characters in narratives in an immersive environment.
Outcome:	The proposed system generates human-like responses guided by personality traits extracted from narratives.

EmoCharacter: Evaluating the Emotional Fidelity of Role-Playing Agents in Dialogues (2025.naacl-long)

Copied to clipboard

Qiming Feng, Qiujie Xie, Xiaolong Wang, Qingqiu Li, Yuejie Zhang, Rui Feng, Tao Zhang, Shang Gao

Dialogue Systems & Conversational Agents

Challenge:	EmoCharacter evaluates emotional fidelity of role-playing agents in dialogues . current evaluations focus on personality fidelity, tone imitation, and knowledge consistency .
Approach:	They propose a benchmark to assess emotional fidelity of role-playing agents in dialogues using large language models.
Outcome:	The proposed benchmark measures emotional fidelity of role-playing agents and the characters they portray.

CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation (2024.acl-long)

Copied to clipboard

Quan Tu, Shilong Fan, Zihang Tian, Tianhao Shen, Shuo Shang, Xin Gao, Rui Yan

Dialogue Systems & Conversational Agents

Challenge:	CharacterEval is a benchmark for comprehensive RPCA assessment in Chinese . authors show that Chinese LLMs exhibit more promising capabilities than GPT-4 in role-playing conversation.
Approach:	They propose a Chinese benchmark for comprehensive RPCA assessment . they use a dataset of Chinese role-playing dialogues and character profiles .
Outcome:	The proposed benchmark demonstrates that Chinese LLMs exhibit more promising capabilities than GPT-4 in Chinese role-playing conversation.

A Framework for Exploring Player Perceptions of LLM-Generated Dialogue in Commercial Video Games (2023.findings-emnlp)

Copied to clipboard

Nader Akoury, Qian Yang, Mohit Iyyer

Dialogue Systems & Conversational Agents

Challenge:	evaluating the player experience in a roleplaying game augmented with LLM-generated dialogue remains a major challenge.
Approach:	They propose a dynamic evaluation framework for the dialogue management systems that govern the task-oriented dialogue often found in roleplaying video games.
Outcome:	The proposed framework directly evaluates the performance of LLM-generated dialogue in a role-playing game with 28 players.

ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities (2025.findings-naacl)

Copied to clipboard

Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, Ruoming Pang

Dialogue Systems & Conversational Agents

Challenge:	Recent advances in large language models have led to a growing interest in tool assisted LLMs . toolSandbox includes stateful tool execution, implicit state dependencies between tools .
Approach:	a new tool-based evaluation tool is released to help LLMs evaluate their tool-use capabilities. a tool-driven evaluation tool includes stateful tool execution, implicit state dependencies between tools and a built-in user simulator.
Outcome:	the toolSandbox evaluation benchmark shows that open source and proprietary models have a performance gap . the benchmarks show that even the most capable LLMs are challenged by state dependent tasks .

CharacterCraft: Bridging the Literature-Reality Dialogue Gap for Practical Role-Playing Agents (2025.findings-emnlp)

Copied to clipboard

Xuyan Yin, Xinran Yang, Zihao Li, Lixin Zou, Chenliang Li

Dialogue Systems & Conversational Agents

Challenge:	Existing dialogue datasets have a bias between query distributions and real-world user language usage.
Approach:	They propose a framework for Chinese role-playing and a robust evaluation method . they propose specialized Chinese dialogue extraction model and specialized memory retrieval module .
Outcome:	The proposed framework extracts character dialogue from novels and ensures high data quality.

MIRAGE: Exploring How Large Language Models Perform in Complex Social Interactive Environments (2025.acl-short)

Copied to clipboard

Yin Cai, Zhouhong Gu, Zhaohan Du, Zheyu Ye, Shaosheng Cao, Yiqian Xu, Hongwei Feng, Ping Chen

Language Models

Challenge:	Large Language Models (LLMs) have shown remarkable capabilities in environmental perception, reasoning-based decision-making, and simulating complex human behaviors, particularly in interactive role-playing contexts.
Approach:	They propose a framework to assess LLMs' proficiency in portraying advanced human behaviors through murder mystery games using eight intricately crafted scripts.
Outcome:	The framework evaluates LLMs' performance in portraying advanced human behaviors through murder mystery games.

Character-LLM: A Trainable Agent for Role-Playing (2023.emnlp-main)

Copied to clipboard

Yunfan Shao, Linyang Li, Junqi Dai, Xipeng Qiu

Challenge:	Large language models (LLMs) can be used to simulate human behaviors . a recent study suggests that LLMs can be more effective at generating human behavior .
Approach:	They propose to use large language models to train agents with the profile, experience, and emotional states of a specific person instead of using limited prompts to instruct ChatGPT API.
Outcome:	The proposed model trains agents with the profile, experience, and emotional states of a specific person instead of using limited prompts to instruct ChatGPT API.