GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing (2026.acl-long)
Copied to clipboard
Ming Wang, Shuang Wu, Bixuan Wang, Lu Lin, Yuxin Chen, Xiaocui Yang, Daling Wang, Shi Feng, Yifei Zhang, Yufan Sun
| Challenge: | Large language models (LLMs) inherit contamination from training corpora, directional bias under social-desirability framing, and limited responsiveness to context beyond the item text. |
| Approach: | They propose a paradigm that reformulates TAT, Rorschach, and SCT with newly generated stimuli and organises assessment as a three-stage pipeline. |
| Outcome: | The proposed paradigm reformulates TAT, Rorschach, and SCT with newly generated stimuli and organises assessment as a three-stage pipeline. |
Similar Papers
ROBBIE: Robust Bias Evaluation of Large Generative Language Models (2023.emnlp-main)
Copied to clipboard
David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi-Yu, Eleonora Presani, Adina Williams, Eric Smith
| Challenge: | generative large language models (LLMs) are becoming more performant and prevalent . we need tools to measure and improve their fairness, authors say . |
| Approach: | They propose to compare 6 different prompt-based bias and toxicity metrics across 12 demographic axes and 5 families of generative large language models. |
| Outcome: | The proposed model can be tested on more datasets to better characterize and mitigate biases . the study compared 6 prompt-based bias and toxicity metrics across 12 demographic axes and 5 families of generative large language models. |
CAPE: Context-Aware Personality Evaluation Framework for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies use a context-free approach to assess humans . existing studies use the Disney World test, which ignores real-world applications . |
| Approach: | They propose a framework to assess personality traits in large language models . they use conversational history to quantify the consistency of LLM responses . |
| Outcome: | The proposed framework improves consistency of responses in large language models . it also shows that conversational history enhances consistency and personality shifts . |
LLM Questionnaire Completion for Automatic Psychiatric Assessment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Psychiatric evaluations are heavily based on patient verbal reports of disturbed feelings, thoughts, behaviors, and their changes over time. |
| Approach: | They employ a Large Language Model to convert unstructured psychological interviews into structured questionnaires spanning various psychiatric and personality domains. |
| Outcome: | The proposed model improves diagnostic accuracy compared to baselines. |
You don’t need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments (2024.naacl-long)
Copied to clipboard
Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Lajanugen Logeswaran, Moontae Lee, Dallas Card, David Jurgens
| Challenge: | Large Language Models (LLMs) are popular for research in social sciences . currently, prompting LLMs is insufficient to accurately and reliably capture model perceptions, and we discuss potential alternatives to improve this. |
| Approach: | They construct a dataset that contains 693 questions encompassing 39 different instruments of persona measurement on 115 persona axes and a set of questions containing minor variations. |
| Outcome: | The proposed model can generate answers and negate statements in a consistent and robust manner. |
On the Reliability of Psychological Scales on Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on examining Large Language Models’ characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics. |
| Approach: | They propose to examine the reliability of personality tests to LLMs by using psychological scales. |
| Outcome: | The proposed model can represent diverse personalities with specific prompt instructions. |
Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics (2025.findings-naacl)
Copied to clipboard
Seungbeen Lee, Seungwon Lim, Seungju Han, Giyeong Oh, Hyungjoo Chae, Jiwan Chung, Minju Kim, Beong-woo Kwak, Yeonsoo Lee, Dongha Lee, Jinyoung Yeo, Youngjae Yu
| Challenge: | Recent advances in Large Language Models (LLMs) have led to their adaptation as conversational agents. |
| Approach: | They propose a new benchmark that uses 8K multi-choice questions to assess the personality of Large Language Models. |
| Outcome: | The proposed personality test outperforms existing personality tests for LLMs in reliability and validity. |
SynthEval: Hybrid Behavioral Testing of NLP Models with Synthetic Evaluation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing frameworks for benchmarking in NLP often overestimate performance . however, manually creating a variety of test types requires significant human labor . |
| Approach: | They propose a framework that leverages large language models to generate a wide range of test types . they first generate sentences via LLMs and then identifies challenging examples . |
| Outcome: | The proposed framework overestimates performance on two classification tasks. |
Improve LLM-as-a-Judge Ability as a General Ability (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies focus on generative judges, but only on their judge ability. |
| Approach: | They propose a method that leverages the generative and reasoning capabilities of large language models to evaluate LLM responses across diverse scenarios, providing accurate preference signals. |
| Outcome: | The proposed model performs on RewardBench with only 2% to 40% of the data required by other training frameworks. |
Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study (2026.acl-long)
Copied to clipboard
| Challenge: | a psychometric framework is proposed to quantify and mitigate socially desirable responding (SDR) in questionnaire-based evaluation of large language models. |
| Approach: | They propose a psychometric framework to quantify and mitigate socially desirable responding (SDR) in questionnaire-based evaluation of large language models. |
| Outcome: | The proposed framework quantifies and mitigates socially desirable responding (SDR) in questionnaire-based evaluation of large language models. |
Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practice (2025.acl-long)
Copied to clipboard
| Challenge: | standardized questionnaires are essential tools for mental health screening, but computational approaches bypass these tools in favor of black-box classification. |
| Approach: | They propose a questionnaire-guided screening framework that bridges psychological practice and computational methods through adaptive Retrieval-Augmented Generation. |
| Outcome: | The proposed framework matches or outperforms state-of-the-art performance on Reddit-based benchmarks and extends to self-harm screening. |