Do Large Language Models Align with Core Mental Health Counseling Competencies? (2025.findings-naacl)
Copied to clipboard
Viet Cuong Nguyen, Mohammad Taher, Dongwan Hong, Vinicius Konkolics Possobom, Vibha Thirunellayi Gopalakrishnan, Ekta Raj, Zihang Li, Heather J. Soled, Michael L. Birnbaum, Srijan Kumar, Munmun De Choudhury
| Challenge: | Large language models are promising for mental health, but their alignment with core counseling competencies remains underexplored. |
| Approach: | They propose a benchmark to evaluate 22 general-purpose and medical-finetuned LLMs across five key competencies. |
| Outcome: | The proposed model outperforms generalist models in Intake, Assessment & Diagnosis but struggles with core counseling attributes and professional practice & ethics. |
Similar Papers
When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation (2026.eacl-long)
Copied to clipboard
Abeer Badawi, Elahe Rahimi, Md Tahmid Rahman Laskar, Sheri Grach, Lindsay Bertrand, Lames Danok, Prathiba Dhanesh, Jimmy Huang, Frank Rudzicz, Elham Dolatabadi
| Challenge: | Existing benchmarks for large language models are limited in scale, authenticity, and reliability due to the emotionally complex nature of therapeutic dialogue. |
| Approach: | They propose two benchmarks that provide a framework for evaluating large language models for mental health support. |
| Outcome: | The proposed framework provides a framework for generation and evaluation of large-scale authentic dialogue datasets and judge-reliability assessments. |
PsychEthicsBench: Evaluating Large Language Models Against Australian Mental Health Ethics (2026.findings-acl)
Copied to clipboard
Yaling Shen, Stephanie Fong, Yiwen Jiang, Zimu Wang, Feilong Tang, Qingyang Xu, Xiangyu Zhao, Zhongxing Xu, Jiahe Liu, Jinpeng Hu, Dominic Dwyer, Zongyuan Ge
| Challenge: | Mental disorders affect nearly one in seven people worldwide, yet the vast majority do not receive adequate care. |
| Approach: | They propose a framework to evaluate LLMs' ethical knowledge and behavioral responses through multiple-choice and open-ended tasks with fine-grained ethicality annotations. |
| Outcome: | Empirical results across 14 models reveal that refusal rates are poor indicators of ethical behavior, revealing a significant divergence between safety triggers and clinical appropriateness. |
Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark (2024.emnlp-main)
Copied to clipboard
Fenglin Liu, Zheng Li, Hongjian Zhou, Qingyu Yin, Jingfeng Yang, Xianfeng Tang, Chen Luo, Ming Zeng, Haoming Jiang, Yifan Gao, Priyanka Nigam, Sreyashi Nag, Bing Yin, Yining Hua, Xuan Zhou, Omid Rohanian, Anshul Thakur, Lei Clifton, David Clifton
| Challenge: | Existing studies focus on evaluating large language models in close-ended QA tasks, but many clinical decisions involve answering open-ended questions without pre-set options. |
| Approach: | They construct a benchmark to better understand large language models in the clinic . they use existing datasets to evaluate LLMs in clinical situations . |
| Outcome: | The proposed model outperforms human experts in multiple medical tasks. |
Can AI Relate: Testing Large Language Model Response for Mental Health Support (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are already being piloted for clinical use in hospitals . recent failures of the Tessa chatbot have led to doubts about their reliability in high-stakes settings. |
| Approach: | They propose safety guidelines for the potential deployment of large language models for mental health response. |
| Outcome: | The proposed framework measures equity in empathy and adherence of LLM responses to motivational interviewing theory. |
Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit remarkable capabilities in various generative tasks, but their adaptation to domain-specific intricacies remains challenging. |
| Approach: | They propose to use a planning engine to orchestrate structuring knowledge alignment to achieve high-order planning by encapsulating domain knowledge and leveraging sheaf convolution learning to enhance its understanding of the dialogue’s structural nuances. |
| Outcome: | The proposed framework improves on existing LLMs and shows that it can generate better summaries with better quality and better execution. |
Can LLMs Reason Like Doctors? Exploring the Limits of Large Language Models in Complex Medical Reasoning (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown remarkable progress in reasoning across multiple domains, but it remains unclear whether their abilities reflect genuine reasoning or sophisticated pattern matching. |
| Approach: | They conduct one of the largest evaluations to date, assessing 77 LLMs . they select three medical question answering (QA) benchmarks targeting reasoning processes . |
| Outcome: | The results highlight the need to improve specific reasoning strategies to better reflect medical decision-making. |
Preference Learning Unlocks LLMs’ Psycho-Counseling Skills (2026.findings-acl)
Copied to clipboard
| Challenge: | Current LLMs struggle to consistently provide effective responses to client speeches due to the lack of supervision from high-quality real psycho-counseling data. |
| Approach: | They propose to use a dataset to evaluate therapists' responses to client speeches using a set of professional and comprehensive principles to evaluate their responses. |
| Outcome: | The proposed model achieves an impressive win rate of 87% against GPT-4o. |
Towards Interpretable Mental Health Analysis with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on large language models lack adequate evaluations and prompting strategies for explainability. |
| Approach: | They evaluate the mental health analysis and emotional reasoning ability of large language models (LLMs) using 11 datasets across 5 tasks. |
| Outcome: | The proposed model shows strong in-context learning ability but still has a significant gap with advanced task-specific methods. |
A Survey of Large Language Models in Psychotherapy: Current Landscape and Future Directions (2025.findings-acl)
Copied to clipboard
Hongbin Na, Yining Hua, Zimu Wang, Tao Shen, Beibei Yu, Lilin Wang, Wei Wang, John Torous, Ling Chen
| Challenge: | Large language models (LLMs) can handle extensive context and multi-turn reasoning. |
| Approach: | They propose a taxonomy dividing psychotherapy into stages of assessment, diagnosis, and treatment to examine LLM advancements and challenges. |
| Outcome: | The proposed taxonomy reveals imbalances in current research, such as a focus on common disorders, linguistic biases, fragmented methods, and limited theoretical integration. |
NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in large language models have demonstrated their strong performance on IQ test questions, achieving high scores across many languages. |
| Approach: | They propose a dataset to evaluate cognitive multimodal reasoning and problem-solving skills of large models. |
| Outcome: | The proposed dataset contains 2,728 multiple-choice questions and 4,642 images spanning 26 categories. |