Beyond Self-Reports: Multi-Observer Agents for Personality Assessment in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Self-report questionnaires are used to assess LLM personality traits, but they fail to capture behavioral nuances due to biases and meta-knowledge contamination. |
| Approach: | They propose a multi-observer framework for personality trait assessments in LLM agents that draws on informant-report methods in psychology. |
| Outcome: | The proposed framework combines multiple observers with a subject LLM agent to assess its Big Five personality traits. |
Similar Papers
Modeling, Evaluating, and Embodying Personality in LLMs: A Survey (2025.findings-emnlp)
Copied to clipboard
Iago Alves Brito, Julia Soares Dollis, Fernanda Bufon Färber, Pedro Schindler Freire Brasil Ribeiro, Rafael Teixeira Sousa, Arlindo Rodrigues Galvão Filho
| Challenge: | This survey provides a comprehensive overview of the LLM-driven personality scenario. |
| Approach: | This survey provides a comprehensive overview of the LLM-driven personality scenario. |
| Outcome: | The proposed taxonomy analyzes the limitations of existing methods and identifies key research gaps. |
PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have shown that LLMs can generate content that aligns with their assigned personality traits, but there is limited research on whether they consistently reflect specific personality traits. |
| Approach: | They propose to study the behavior of LLM-based agents which they refer to as LLM personas and simulate them to measure their personality traits. |
| Outcome: | The proposed model is based on the Big Five personality model and has been validated by human evaluations and automatic evaluations. |
Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation (2026.acl-long)
Copied to clipboard
Jiaju Chen, Yuxuan Lu, Xiaojie Wang, Huimin Zeng, Jing Huang, Jiri Gesi, Ying Xu, Bingsheng Yao, Dakuo Wang
| Challenge: | Existing "LLM-as-a-judge" evaluation frameworks are limited by persona descriptions and are not generalizable to other tasks. |
| Approach: | They propose a framework that can automatically construct multiple evaluator personas with distinct dimensions from relevant text documents and instantiate LLM agents with the persona. |
| Outcome: | The proposed framework can believably simulate human evaluators . it extracts stakeholders' diverse perspectives from the provided research papers and constructs personas for the agents . |
Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics (2025.findings-naacl)
Copied to clipboard
Seungbeen Lee, Seungwon Lim, Seungju Han, Giyeong Oh, Hyungjoo Chae, Jiwan Chung, Minju Kim, Beong-woo Kwak, Yeonsoo Lee, Dongha Lee, Jinyoung Yeo, Youngjae Yu
| Challenge: | Recent advances in Large Language Models (LLMs) have led to their adaptation as conversational agents. |
| Approach: | They propose a new benchmark that uses 8K multi-choice questions to assess the personality of Large Language Models. |
| Outcome: | The proposed personality test outperforms existing personality tests for LLMs in reliability and validity. |
Can LLM Agents Maintain a Persona in Discourse? (2025.emnlp-main)
Copied to clipboard
Pranav Bhandari, Nicolas Fay, Michael J Wise, Amitava Datta, Stephanie Meek, Usman Naseem, Mehwish Nasim
| Challenge: | Large language models are often subjected to context-shifting behaviour, resulting in a lack of consistent and interpretable personality-aligned interactions. |
| Approach: | They propose to use two conversation agents to generate a discourse with an assigned personality from the OCEAN framework and then use multiple judge agents to infer original traits. |
| Outcome: | The proposed model is based on two conversation agents with a personality assigned from the OCEAN framework and then multiple judge agents to infer the original traits assigned. |
Can ChatGPT Assess Human Personalities? A General Evaluation Framework (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies study the virtual personalities of LLMs but rarely explore the possibility of analyzing human personalities via LLM. |
| Approach: | They propose to use Myers–Briggs Type Indicator (MBTI) tests to generate unbiased prompts and replace the subject in question statements to enable flexible queries and assessments. |
| Outcome: | The proposed framework enables LLMs to flexibly assess personalities of different groups of people. |
PersonaGym: Evaluating Persona Agents and LLMs (2025.findings-emnlp)
Copied to clipboard
Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik R Narasimhan, Vishvak Murahari
| Challenge: | Persona agents are LLM agents conditioned to act according to an assigned persona . evaluating how faithfully these agents adhere to their personas remains a challenge . |
| Approach: | a new study evaluates persona agents' ability to act according to an assigned persona . a persona agent's person score is a human-aligned automatic metric that can be used to evaluate a model . |
| Outcome: | a new evaluation framework and a human-aligned automatic metric show that persona agents can perform better. |
Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing literature on leveraging persona in large language models is disorganized and lacks a systematic taxonomy . leveraging peopleas has resurfaced as an ideal lens for adapting LLMs for specific contexts . |
| Approach: | They propose to categorize current research on leveraging persona in large language models . they propose to use a comprehensive survey to categorize existing studies . |
| Outcome: | The proposed framework is a promising framework for tailoring large language models to specific contexts. |
PADO: Personality-induced multi-Agents for Detecting OCEAN in human-generated texts (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for personality detection are limited due to the latent and relative nature of personality and lack of annotated datasets. |
| Approach: | They propose a method that exploits the inherent knowledge of Large Language Models to capture the relative nature of personality traits by comparing contrasting perspectives. |
| Outcome: | The proposed approach exploits the inherent knowledge of Large Language Models to capture the relative nature of personality traits. |
Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplification and Resistance in Multi-Agent Based LLM-as-Judge (2025.findings-emnlp)
Copied to clipboard
Chiyu Ma, Enpei Zhang, Yilun Zhao, Wenjun Liu, Yaning Jia, Peijun Qing, Lin Shi, Arman Cohan, Yujun Yan, Soroush Vosoughi
| Challenge: | LLM-as-Judge frameworks provide scalable alternative to human evaluation . but the question of how intrinsic biases manifest in these settings remains unexplored . |
| Approach: | They conduct systematic analysis of four bias types in multi-agent LLM-as-Judge frameworks . they find debate framework amplifies biases sharply after initial debate . |
| Outcome: | The proposed frameworks amplify biases after debate and show they are stronger in meta-judge scenarios. |