RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models (2024.findings-acl)
Copied to clipboard
Noah Wang, Z.y. Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang, Jie Fu, Junran Peng
| Challenge: | Large Language Models (LLMs) have paved the way for complex tasks such as role-playing. |
| Approach: | They propose a framework to benchmark, elicit, and enhance role-playing abilities in Large Language Models. |
| Outcome: | The proposed framework improves role-playing abilities with 168,093 samples. |
Similar Papers
RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing (2025.findings-emnlp)
Copied to clipboard
Hao Xiang, Tianyi Tang, Yang Su, Bowen Yu, An Yang, Fei Huang, Yichang Zhang, Yaojie Lu, Hongyu Lin, Xianpei Han, Jingren Zhou, Junyang Lin, Le Sun
| Challenge: | Existing benchmarks focus on character-centric approach and fail to reflect real-world applications. |
| Approach: | RMTBench is a user-centric bilingual role-playing benchmark featuring 80 diverse characters and over 8,000 dialogue rounds. |
| Outcome: | RMTBench features 80 diverse characters and over 8,000 dialogue rounds. |
RoleMRC: A Fine-Grained Composite Benchmark for Role-Playing and Instruction-Following (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing role-playing datasets mostly contribute to controlling role style and knowledge boundaries, but overlook role-following in instruction-follower scenarios. |
| Approach: | They propose a fine-grained role-playing and instruction-following composite benchmark, named RoleMRC, which includes multi-turn dialogues between ideal roles and humans, including free chats or discussions upon given passages . |
| Outcome: | The proposed model improves instruction-following without compromising general role-playing and reasoning capabilities. |
Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment (2024.acl-long)
Copied to clipboard
| Challenge: | Existing work cheaply emulates LLMs, allowing users to create profiles for their preferred characters. |
| Approach: | They propose a self-alignment method that encourages an instruction-following LLM to simulate role-play dialogues as a variant of reading comprehension. |
| Outcome: | The proposed model outperforms open-source role-play benchmarks and the roleplay subset of MT-Bench in multiple parameters. |
LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient (2026.acl-long)
Copied to clipboard
Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Using generic and efficient benchmark generators, human annotators are limited by inefficiency . current benchmark generator methods rely on seed signals, leading to long cycles and high costs . |
| Approach: | They propose a framework to evaluate LLMs as generic benchmark generators and integrate them as BenchMaker. |
| Outcome: | The proposed framework achieves comparable performance to human-annotated benchmarks on most metrics. |
Can Role Vectors Affect LLM Behaviour? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study has shown that personas influence LLM performance, but their direct impact remains unclear. |
| Approach: | They propose a novel approach to guiding LLM behaviour through role vectors . they construct 29 role vector derived from model activations and evaluate their impact . |
| Outcome: | The proposed approach improves in-domain task performance while yielding unexpected gains. |
Crab: A Novel Configurable Role-Playing LLM with Assessing Benchmark (2025.acl-long)
Copied to clipboard
Kai He, Yucheng Huang, Wenqing Wang, Delong Ran, Dongming Sheng, Junxuan Huang, Qika Lin, Jiaxing Xu, Wenqiang Liu, Mengling Feng
| Challenge: | Existing RP-LLMs employ only a single role with numerous dialogues, but Crab enables dynamic configuration of desired roles, thereby enhancing related flexibility and adaptability. |
| Approach: | They propose a Configurable Role-Playing LLM with Assessing Benchmark that combines a Role dataset curation, persona-emodying Llm construction, and comprehensive benchmark creation for RP dialogue generation. |
| Outcome: | The proposed model outperforms existing LLMs in performing fine-grained evaluations of RP while keeping dialogue per role minimal. |
GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing LLM leaderboards often reference scores reported in other papers without consistent settings and prompts, which may encourage cherry-picking favored settings and for better results. |
| Approach: | They propose an open-source and reproducible LLM evaluation suite built on top of OpenAI Evals that systematically evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
| Outcome: | The evaluation suite is built on top of OpenAI Evals and evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
VoiceBench: Benchmarking LLM-Based Voice Assistants (2026.tacl-1)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have enabled real-time speech interactions through LLMs. |
| Approach: | They propose a benchmark specifically designed to assess LLM-based voice assistants. |
| Outcome: | The proposed benchmark measures the performance of LLM-based voice assistants across eight tasks. |
Measuring and Benchmarking Large Language Models’ Capabilities to Generate Persuasive Language (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on specific domains or types of persuasion, but a general study has focused on how LLMs produce persuasive text. |
| Approach: | They construct a dataset to measure and benchmark the ability of Large Language Models (LLMs) to produce persuasive text. |
| Outcome: | The proposed model can be used to generate persuasive text across domains and domains. |
HER: Human-like Reasoning and Reinforcement Learning for LLM Role-playing (2026.findings-acl)
Copied to clipboard
Chengyu Du, Xintao Wang, Aili Chen, Weiyuan Li, Rui Xu, Junteng Liu, Zishan Huang, Rong Tian, Zijun Sun, Yuhao Li, Liheng Feng, Deming Ding, Pengyu Zhao, Yanghua Xiao
| Challenge: | Existing models for LLM role-playing lack high-quality datasets with explicit reasoning traces and reliable reward signals aligned with human preferences. |
| Approach: | They propose a unified framework for cognitive-level persona simulation that strictly distinguishes characters’ first-person thinking processes from LLMs’ third-person reasoning. |
| Outcome: | The proposed framework outperforms the Qwen3-32B baseline model and achieves a 30.26% and 14.97% performance on the minimax benchmarks. |