MedQA-CS: Objective Structured Clinical Examination (OSCE)-Style Benchmark for Evaluating LLM Clinical Skills

Challenge:	Current clinical LLM benchmarks fail to evaluate advanced clinical skills in AI and large language models (LLMs).
Approach:	They propose a framework to evaluate large language models (LLMs) using two instruction-following tasks designed to reflect real clinical scenarios.
Outcome:	The proposed framework evaluates LLMs through two instruction-following tasks designed to reflect real clinical scenarios.

Similar Papers

Overview

Challenge

Approach

Outcome

LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation (2025.findings-emnlp)

Copied to clipboard

Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang

Language Models

Challenge:	Current medical benchmarks have limitations in question design, data sources and evaluation methods.
Approach:	They propose a new benchmark covering five core medical areas . it includes 2,996 questions created from real-world electronic health records .
Outcome:	The proposed model covers five core medical areas and includes 2,996 questions created from real-world electronic health records and expert-designed clinical scenarios.

Automatic Evaluation of Healthcare LLMs Beyond Question-Answering (2025.naacl-short)

Copied to clipboard

Anna Arias-Duart, Pablo Agustin Martin-Torres, Daniel Hinjos, Pablo Bernabeu-Perez, Lucia Urcelay Ganzabal, Marta Gonzalez Mallo, Ashwin Kumar Gururajan, Enrique Lopez-Cuena, Sergio Alvarez-Napagao, Dario Garcia-Gasulla

Question Answering

Challenge:	Current Large Language Models (LLMs) benchmarks are often based on open-ended or close-ended QA evaluations, avoiding the requirement of human labor.
Approach:	They propose a multi-axis suite for healthcare LLM evaluation, exploring correlations between open and close benchmarks and metrics.
Outcome:	The proposed framework explores correlations between open and close benchmarks and metrics in the healthcare domain, with blind spots and overlaps in existing methodologies.

From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations (2025.emnlp-main)

Copied to clipboard

Benlu Wang, Iris Xia, Yifan Zhang, Junda Wang, null Feiyun Ouyang, Shuo Han, Arman Cohan, Hong Yu, Zonghai Yao

Reasoning

Challenge:	Existing benchmarks assess only the final answer with a wide numerical tolerance, overlooking systematic reasoning failures and potentially causing serious clinical misjudgments.
Approach:	They propose a new step-by-step evaluation pipeline that assesses formula selection, entity extraction, and arithmetic computation.
Outcome:	The proposed method improves the accuracy of large language models on medical benchmarks from 16.35% to 53.19%.

ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models (2026.findings-acl)

Copied to clipboard

Ruihui Hou, Siyi Zhu, Ziyue Huai, Guangya Yu, Yongqi Fan, ChunMing Wang, Tong Ruan

Semantic Text Processing

Challenge:	Existing benchmarks assess LLM performance in single-course settings and lack systematic evaluation in multi-course scenarios, where a patient’s condition evolves over time.
Approach:	They propose to use large language models to assess their performance in multi-course clinical decision-making scenarios where a patient’s condition evolves over time.
Outcome:	The proposed model includes 1,275 Chinese and 5,804 English samples across four stages from admission to discharge.

Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models (2026.acl-long)

Copied to clipboard

Wenxuan Wang, Zizhan Ma, Guo Yu, Yiu-Fai Cheung, Meidan Ding, Jie Liu, Wenting Chen, Linlin Shen

Language Models

Challenge:	Large language models (LLMs) are proving significant potential in healthcare, prompting numerous benchmarks to evaluate their capabilities.
Approach:	They propose a framework that deconstructs benchmark development into five stages from design to governance and provides a checklist of 46 medically-tailored criteria.
Outcome:	The framework deconstructs benchmark development into five stages from design to governance and provides a comprehensive checklist of 46 medically-tailored criteria.

What Does Infect Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs (2026.eacl-long)

Copied to clipboard

Xinlan Yan, Di Wu, Yibin Lei, Christof Monz, Iacer Calixto

Question Answering

Challenge:	S-MedQA is an English question-answering dataset designed for benchmarking large language models in fine-grained clinical specialties.
Approach:	They propose to use an English medical question-answering dataset to benchmark large language models in clinical specialties.
Outcome:	The proposed dataset is designed to benchmark large language models in medical specialties.

Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark (2024.emnlp-main)

Copied to clipboard

Fenglin Liu, Zheng Li, Hongjian Zhou, Qingyu Yin, Jingfeng Yang, Xianfeng Tang, Chen Luo, Ming Zeng, Haoming Jiang, Yifan Gao, Priyanka Nigam, Sreyashi Nag, Bing Yin, Yining Hua, Xuan Zhou, Omid Rohanian, Anshul Thakur, Lei Clifton, David Clifton

Language Models

Challenge:	Existing studies focus on evaluating large language models in close-ended QA tasks, but many clinical decisions involve answering open-ended questions without pre-set options.
Approach:	They construct a benchmark to better understand large language models in the clinic . they use existing datasets to evaluate LLMs in clinical situations .
Outcome:	The proposed model outperforms human experts in multiple medical tasks.

CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios (2024.emnlp-main)

Copied to clipboard

Zetian Ouyang, Yishuai Qiu, Linlin Wang, Gerard De Melo, Ya Zhang, Yanfeng Wang, Liang He

Language Models

Challenge:	Chinese medical large language models (LLMs) are underperforming on this benchmark, especially where medical reasoning and factual consistency are vital.
Approach:	They propose a benchmark with 14 expert-guided clinical scenarios to assess the medical ability of large language models across 7 pivot dimensions.
Outcome:	The proposed benchmark has been validated in several ways.

CMedCalc-Bench: A Fine-Grained Benchmark for Chinese Medical Calculations in LLM (2025.emnlp-main)

Copied to clipboard

Yunyan Zhang, Zhihong Zhu, Xian Wu

Reasoning

Challenge:	Existing medical NLP benchmarks focus on qualitative reasoning and textual comprehension, but lack of fine-grained evaluation of intermediate reasoning.
Approach:	They propose a Chinese medical calculation benchmark that disentangles clinical entity extraction from numerical computation.
Outcome:	The proposed framework disentangles clinical entity extraction from numerical computation, enabling systematic diagnosis of model deficiencies.

Benchmarking LLMs on Authentic Cases from Medical Journals (2026.findings-acl)

Copied to clipboard

Wanlong Liu, Junying Chen, Yunjin Yang, Prayag Tiwari, Wenyu Chen, Benyou Wang

Challenge:	Existing medical benchmarks suffer from performance saturation due to medical exam questions.
Approach:	They evaluate the performance of over 20 open-source and proprietary large language models and benchmark them against human medical experts.
Outcome:	The new benchmark is based on authentic clinical cases sourced from medical journals and implements rigorous human review process to ensure the quality and reliability of the benchmark.