Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions (2025.naacl-long)
Copied to clipboard
| Challenge: | Medical board exams or general clinical questions do not capture the complexity of real clinical cases. |
| Approach: | They construct two datasets that are structured as multiple-choice question-answering tasks accompanied by expert-written explanations. |
| Outcome: | The proposed datasets are harder than previous benchmarks. |
Similar Papers
Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark (2024.emnlp-main)
Copied to clipboard
Fenglin Liu, Zheng Li, Hongjian Zhou, Qingyu Yin, Jingfeng Yang, Xianfeng Tang, Chen Luo, Ming Zeng, Haoming Jiang, Yifan Gao, Priyanka Nigam, Sreyashi Nag, Bing Yin, Yining Hua, Xuan Zhou, Omid Rohanian, Anshul Thakur, Lei Clifton, David Clifton
| Challenge: | Existing studies focus on evaluating large language models in close-ended QA tasks, but many clinical decisions involve answering open-ended questions without pre-set options. |
| Approach: | They construct a benchmark to better understand large language models in the clinic . they use existing datasets to evaluate LLMs in clinical situations . |
| Outcome: | The proposed model outperforms human experts in multiple medical tasks. |
Can LLMs Reason Like Doctors? Exploring the Limits of Large Language Models in Complex Medical Reasoning (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown remarkable progress in reasoning across multiple domains, but it remains unclear whether their abilities reflect genuine reasoning or sophisticated pattern matching. |
| Approach: | They conduct one of the largest evaluations to date, assessing 77 LLMs . they select three medical question answering (QA) benchmarks targeting reasoning processes . |
| Outcome: | The results highlight the need to improve specific reasoning strategies to better reflect medical decision-making. |
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation (2025.findings-emnlp)
Copied to clipboard
Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Current medical benchmarks have limitations in question design, data sources and evaluation methods. |
| Approach: | They propose a new benchmark covering five core medical areas . it includes 2,996 questions created from real-world electronic health records . |
| Outcome: | The proposed model covers five core medical areas and includes 2,996 questions created from real-world electronic health records and expert-designed clinical scenarios. |
CasiMedicos-Arg: A Medical Question Answering Dataset Annotated with Explanatory Argumentative Structures (2024.emnlp-main)
Copied to clipboard
Ekaterina Sviridova, Anar Yeginbergen, Ainara Estarrona, Elena Cabrio, Serena Villata, Rodrigo Agerri
| Challenge: | Existing tools to aid residents in teaching medical doctors to explain decisions are a key objective of AI in education. |
| Approach: | They present a multilingual dataset for Medical Question Answering where doctors can annotate correct and incorrect diagnoses with argument components and argument relations. |
| Outcome: | The proposed dataset consists of 558 clinical cases with explanations in English, Spanish, French, Italian and annotated with argument components and argument relations. |
ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing explanation datasets for large language models are limited to the English language and general domain, leading to a scarcity of linguistic diversity and a lack of resources in specialized domains, such as medical. |
| Approach: | They propose to use a medical dataset to assess the interpretability of Large Language Models (LLMs) . they propose to analyze medical text and generate rationales for their decisions . |
| Outcome: | The proposed model passes the pharmacist examination with a 75.7% accuracy, while other models like ChatGPT fail. |
Benchmarking LLMs on Authentic Cases from Medical Journals (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing medical benchmarks suffer from performance saturation due to medical exam questions. |
| Approach: | They evaluate the performance of over 20 open-source and proprietary large language models and benchmark them against human medical experts. |
| Outcome: | The new benchmark is based on authentic clinical cases sourced from medical journals and implements rigorous human review process to ensure the quality and reliability of the benchmark. |
AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset (2025.acl-long)
Copied to clipboard
Charles Nimo, Tobi Olatunji, Abraham Toluwase Owodunni, Tassallah Abdullahi, Emmanuel Ayodele, Mardhiyah Sanni, Ezinwanne C. Aka, Folafunmi Omofoye, Foutse Yuehgoh, Timothy Faniran, Bonaventure F. P. Dossou, Moshood O. Yekini, Jonas Kemp, Katherine A Heller, Jude Chidubem Omeke, Chidi Asuzu Md, Naome A Etori, Aïmérou Ndiaye, Ifeoma Okoh, Evans Doe Ocansey, Wendy Kinara, Michael L. Best, Irfan Essa, Stephen Edward Moore, Chris Fourie, Mercy Nyamewaa Asiedu
| Challenge: | Recent advances in large language models (LLMs) performance on medical multiplechoice question (MCQ) benchmarks have stimulated interest from healthcare providers and patients globally. |
| Approach: | They introduce AfriMed-QA, the first largescale Pan-African English multi-specialty medical Question-Answering (QA) dataset, with 15,000 questions sourced from over 60 medical schools across 16 countries. |
| Outcome: | The proposed model outperforms other models in the medical field and is compared with other models. |
DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models (2026.findings-acl)
Copied to clipboard
Yakun Zhu, Zhongzhen Huang, Linjie Mu, Yutong Huang, Wei Nie, Jiaji Liu, Shaoting Zhang, Pengfei Liu, Xiaofan Zhang
| Challenge: | Existing medical benchmarks for diagnostic reasoning are limited in their ability to perform complex tasks. |
| Approach: | They propose to benchmark diagnostic capabilities of large language models to assess their accuracy and generalization bottlenecks. |
| Outcome: | The proposed model achieves 45.82%, 31.09%, and 17.79% accuracy, compared to current models, o3-mini, e1 and DeepSeek-R1 . |
Evaluating Large Vision Language Models on Bangla Medical Visual Question Answering (2026.findings-acl)
Copied to clipboard
Rafid Ahmed, Intesar Tahmid, Mir Sazzat Hossain, Tasnimul Hossain Tomal, Md Mahir Jawad, Anam Borhan Uddin, Md Fahim, Md Farhad Alam Bhuiyan
| Challenge: | Recent advances in Large Language Models and Large Vision Language Model (LVLMs) have demonstrated promising capabilities in complex reasoning tasks, but low-resource contexts like Bangla are underexplored. |
| Approach: | They propose a multilingual medical visual question answering dataset using Bangla. |
| Outcome: | The proposed model performs well on generalized visual tasks but struggles with fine-grained diagnostic reasoning, achieving low accuracy in specialized categories. |
What Does Infect Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | S-MedQA is an English question-answering dataset designed for benchmarking large language models in fine-grained clinical specialties. |
| Approach: | They propose to use an English medical question-answering dataset to benchmark large language models in clinical specialties. |
| Outcome: | The proposed dataset is designed to benchmark large language models in medical specialties. |