Systematic Evaluation of Auto-Encoding and Large Language Model Representations for Capturing Author States and Traits (2025.findings-acl)
Copied to clipboard
Khushboo Singh, Vasudha Varadarajan, Adithya V Ganesan, August Håkan Nilsson, Nikita Soni, Syeda Mahwish, Pranav Chitale, Ryan L. Boyd, Lyle Ungar, Richard N Rosenthal, H. Schwartz
| Challenge: | Large Language Models (LLMs) are increasingly used in human-centered applications, yet their ability to model diverse psychological constructs is not well understood. |
| Approach: | They evaluated a range of Transformer-LMs to predict psychological variables across five major dimensions: affect, substance use, mental health, sociodemographics, and personality. |
| Outcome: | The models predict affect, substance use, mental health, sociodemographics, and personality across five major dimensions. |
Similar Papers
Modeling, Evaluating, and Embodying Personality in LLMs: A Survey (2025.findings-emnlp)
Copied to clipboard
Iago Alves Brito, Julia Soares Dollis, Fernanda Bufon Färber, Pedro Schindler Freire Brasil Ribeiro, Rafael Teixeira Sousa, Arlindo Rodrigues Galvão Filho
| Challenge: | This survey provides a comprehensive overview of the LLM-driven personality scenario. |
| Approach: | This survey provides a comprehensive overview of the LLM-driven personality scenario. |
| Outcome: | The proposed taxonomy analyzes the limitations of existing methods and identifies key research gaps. |
On the Reliability of Psychological Scales on Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on examining Large Language Models’ characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics. |
| Approach: | They propose to examine the reliability of personality tests to LLMs by using psychological scales. |
| Outcome: | The proposed model can represent diverse personalities with specific prompt instructions. |
Evaluating Psychological Safety of Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | a recent study evaluated the psychological safety of large language models. |
| Approach: | They designed unbiased prompts to evaluate the psychological safety of large language models. |
| Outcome: | The proposed prompts showed that they were fine-tuned with behavioral metrics to reduce toxicity. |
PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have shown that LLMs can generate content that aligns with their assigned personality traits, but there is limited research on whether they consistently reflect specific personality traits. |
| Approach: | They propose to study the behavior of LLM-based agents which they refer to as LLM personas and simulate them to measure their personality traits. |
| Outcome: | The proposed model is based on the Big Five personality model and has been validated by human evaluations and automatic evaluations. |
Exploring Large Language Models for Detecting Mental Disorders (2025.emnlp-main)
Copied to clipboard
| Challenge: | Detecting mental disorders and patient emotions through text analysis and machine learning is of increasing interest to researchers over the past decade. |
| Approach: | They compare the performance of traditional machine learning methods and encoder-based models on Russian-language datasets to those of large language models. |
| Outcome: | The proposed models outperform traditional methods on small and noisy datasets, but can perform comparable to language models when trained on patients with clinically confirmed depression. |
How Controllable Are Large Language Models? A Unified Evaluation across Behavioral Granularities (2026.acl-long)
Copied to clipboard
Ziwen Xu, Kewei Xu, Haoming Xu, Haiwen Hong, Longtao Huang, Hui Xue, Ningyu Zhang, Yongliang Shen, Guozhou Zheng, Huajun Chen, Shumin Deng
| Challenge: | Large language models (LLMs) are increasingly deployed in socially sensitive domains, yet their unpredictable behaviors pose significant risks. |
| Approach: | They propose a hierarchical benchmark for evaluating LLM controllability across three domains: language features, sentiment, and personality. |
| Outcome: | The proposed framework offers a principled and interpretable framework for safe and controllable LLM behavior serving as a foundation for future research. |
A Survey of Large Language Models in Psychotherapy: Current Landscape and Future Directions (2025.findings-acl)
Copied to clipboard
Hongbin Na, Yining Hua, Zimu Wang, Tao Shen, Beibei Yu, Lilin Wang, Wei Wang, John Torous, Ling Chen
| Challenge: | Large language models (LLMs) can handle extensive context and multi-turn reasoning. |
| Approach: | They propose a taxonomy dividing psychotherapy into stages of assessment, diagnosis, and treatment to examine LLM advancements and challenges. |
| Outcome: | The proposed taxonomy reveals imbalances in current research, such as a focus on common disorders, linguistic biases, fragmented methods, and limited theoretical integration. |
Generative Psycho-Lexical Approach for Constructing Value Systems in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have raised concerns regarding their intrinsic values. |
| Approach: | They propose a psychologically grounded five-factor value system for Large Language Models that integrates psychological principles with cutting-edge AI priorities. |
| Outcome: | The proposed value system meets standard psychological criteria, improves LLM safety prediction, and enhances Llm alignment, when compared to the canonical Schwartz’s values. |
Are Large Language Models (LLMs) Good Social Predictors? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies suggest that Large Language Models can generate human-like responses, but it is unclear how well they work and where the plausible predictions derive from. |
| Approach: | They propose to use LLMs to generate human-like responses by mutability and accessibility of social inputs to perform a social prediction task. |
| Outcome: | The proposed model performs well in three realistic settings and a novel social prediction task. |
Discovering Language Model Behaviors with Model-Written Evaluations (2023.findings-acl)
Copied to clipboard
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan
| Challenge: | Prior work creates evaluations with crowdwork or existing data sources, which are not always available. |
| Approach: | They generate evaluations automatically with language models (LMs) using crowdwork or existing data sources to find out how they behave . |
| Outcome: | The results show that large LMs repeat back a dialog user’s preferred answer and express greater desire to pursue concerning goals like resource acquisition and goal preservation. |