From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations (2025.emnlp-main)
Copied to clipboard
Benlu Wang, Iris Xia, Yifan Zhang, Junda Wang, null Feiyun Ouyang, Shuo Han, Arman Cohan, Hong Yu, Zonghai Yao
| Challenge: | Existing benchmarks assess only the final answer with a wide numerical tolerance, overlooking systematic reasoning failures and potentially causing serious clinical misjudgments. |
| Approach: | They propose a new step-by-step evaluation pipeline that assesses formula selection, entity extraction, and arithmetic computation. |
| Outcome: | The proposed method improves the accuracy of large language models on medical benchmarks from 16.35% to 53.19%. |
Similar Papers
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation (2025.findings-emnlp)
Copied to clipboard
Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Current medical benchmarks have limitations in question design, data sources and evaluation methods. |
| Approach: | They propose a new benchmark covering five core medical areas . it includes 2,996 questions created from real-world electronic health records . |
| Outcome: | The proposed model covers five core medical areas and includes 2,996 questions created from real-world electronic health records and expert-designed clinical scenarios. |
Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are proving significant potential in healthcare, prompting numerous benchmarks to evaluate their capabilities. |
| Approach: | They propose a framework that deconstructs benchmark development into five stages from design to governance and provides a checklist of 46 medically-tailored criteria. |
| Outcome: | The framework deconstructs benchmark development into five stages from design to governance and provides a comprehensive checklist of 46 medically-tailored criteria. |
Benchmarking LLMs on Authentic Cases from Medical Journals (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing medical benchmarks suffer from performance saturation due to medical exam questions. |
| Approach: | They evaluate the performance of over 20 open-source and proprietary large language models and benchmark them against human medical experts. |
| Outcome: | The new benchmark is based on authentic clinical cases sourced from medical journals and implements rigorous human review process to ensure the quality and reliability of the benchmark. |
A Comprehensive Survey on the Trustworthiness of Large Language Models in Healthcare (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a survey of large language models in healthcare raises critical concerns around trustworthiness . trustworthy of LLMs in healthcare remains underexplored, lacking a systematic review . |
| Approach: | a new survey examines the trustworthiness of large language models in healthcare . a review examines how each dimension affects reliability and ethical deployment of LLMs . |
| Outcome: | The present study examines the trustworthiness of large language models in healthcare . it identifies key gaps in existing approaches and challenges posed by evolving paradigms . |
Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark (2024.emnlp-main)
Copied to clipboard
Fenglin Liu, Zheng Li, Hongjian Zhou, Qingyu Yin, Jingfeng Yang, Xianfeng Tang, Chen Luo, Ming Zeng, Haoming Jiang, Yifan Gao, Priyanka Nigam, Sreyashi Nag, Bing Yin, Yining Hua, Xuan Zhou, Omid Rohanian, Anshul Thakur, Lei Clifton, David Clifton
| Challenge: | Existing studies focus on evaluating large language models in close-ended QA tasks, but many clinical decisions involve answering open-ended questions without pre-set options. |
| Approach: | They construct a benchmark to better understand large language models in the clinic . they use existing datasets to evaluate LLMs in clinical situations . |
| Outcome: | The proposed model outperforms human experts in multiple medical tasks. |
CMedCalc-Bench: A Fine-Grained Benchmark for Chinese Medical Calculations in LLM (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing medical NLP benchmarks focus on qualitative reasoning and textual comprehension, but lack of fine-grained evaluation of intermediate reasoning. |
| Approach: | They propose a Chinese medical calculation benchmark that disentangles clinical entity extraction from numerical computation. |
| Outcome: | The proposed framework disentangles clinical entity extraction from numerical computation, enabling systematic diagnosis of model deficiencies. |
CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios (2024.emnlp-main)
Copied to clipboard
| Challenge: | Chinese medical large language models (LLMs) are underperforming on this benchmark, especially where medical reasoning and factual consistency are vital. |
| Approach: | They propose a benchmark with 14 expert-guided clinical scenarios to assess the medical ability of large language models across 7 pivot dimensions. |
| Outcome: | The proposed benchmark has been validated in several ways. |
MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (2025.findings-acl)
Copied to clipboard
| Challenge: | Several studies have shown that large language models can answer medical questions correctly, outperforming the average human score in some medical exams. |
| Approach: | They introduce MEDEC, the first publicly available benchmark for medical error detection and correction in clinical notes. |
| Outcome: | The proposed model outperforms medical doctors in errors detection and correction tasks. |
ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks assess LLM performance in single-course settings and lack systematic evaluation in multi-course scenarios, where a patient’s condition evolves over time. |
| Approach: | They propose to use large language models to assess their performance in multi-course clinical decision-making scenarios where a patient’s condition evolves over time. |
| Outcome: | The proposed model includes 1,275 Chinese and 5,804 English samples across four stages from admission to discharge. |
EMPEC: A Comprehensive Benchmark for Evaluating Large Language Models Across Diverse Healthcare Professions (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) show their potential in accurately answering biomedical questions, yet current healthcare benchmarks primarily assess knowledge mastered by medical doctors, neglecting other essential professions. |
| Approach: | They evaluated 17 LLMs including proprietary and open-source models and found they struggled with specialized fields and alternative medicine. |
| Outcome: | The examinations for medical PErsonnel in Chinese (EMPEC) features 157,803 exam questions across 124 subjects and 20 healthcare professions. |