Reasoning or Knowledge: Stratified Evaluation of Biomedical LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Medical reasoning in large language models is a complex cognitive process through which clinicians interpret patient data and make diagnostic and therapeutic decisions. |
| Approach: | They propose an evaluation framework that disentangles knowledge recall from reasoning by training a PubMedBERT-based classifier and applying it to 11 widely used biomedical QA benchmarks. |
| Outcome: | The proposed evaluation framework disentangles knowledge recall from reasoning by training a PubMedBERT-based classifier and applying it to 11 widely used biomedical QA benchmarks. |
Similar Papers
MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models can encode knowledge during pre-training on large text corpora, enabling downstream tasks like question answering (QA). |
| Approach: | They construct a dataset derived from systematic reviews to examine their ability to encode medical knowledge and their recall. |
| Outcome: | The proposed model performs well on the biomedical QA dataset. |
Facts Fade Fast: Evaluating Memorization of Outdated Medical Knowledge in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLMs encode extensive knowledge within their parameters, but the knowledge in LLM models can become outdated over time. |
| Approach: | They propose two new LLMs that provide outdated medical advice . they compare the models with a set of QA pairs whose verdict changed through time . |
| Outcome: | The proposed models exhibit memorization of outdated knowledge to some extent. |
Towards Medical Complex Reasoning with LLMs through Medical Verifiable Problems (2025.findings-acl)
Copied to clipboard
| Challenge: | OpenAI o1 has been a significant milestone in large language model development . however, most research in reasoning has focused on mathematical tasks . medical domains require robust reasoning to provide reliable answers . |
| Approach: | They propose a method to verify medical reasoning using a medical verifier . they also propose RL and reinforcement learning to enhance reasoning . |
| Outcome: | The proposed method outperforms general and medical-specific baselines using only 40K verifiable problems. |
Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are often evaluated using multiple-choice questions (MCQs) modeled on exams like the USMLE. |
| Approach: | They created a fictional medical benchmark centered on an imaginary organ, the Glianorex, to separate memorized knowledge from reasoning ability. |
| Outcome: | The proposed model outperforms base models in English but not in French. |
From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations (2025.emnlp-main)
Copied to clipboard
Benlu Wang, Iris Xia, Yifan Zhang, Junda Wang, null Feiyun Ouyang, Shuo Han, Arman Cohan, Hong Yu, Zonghai Yao
| Challenge: | Existing benchmarks assess only the final answer with a wide numerical tolerance, overlooking systematic reasoning failures and potentially causing serious clinical misjudgments. |
| Approach: | They propose a new step-by-step evaluation pipeline that assesses formula selection, entity extraction, and arithmetic computation. |
| Outcome: | The proposed method improves the accuracy of large language models on medical benchmarks from 16.35% to 53.19%. |
PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to QA provide inaccurate answers but lack mechanisms to iteratively refine poor queries. |
| Approach: | They propose a biomedical question answering agent that performs self-critic query refinement . they propose re-reflection methods that kick in only after full retrieval is completed . |
| Outcome: | a biomedical question answering agent achieves 78.32% accuracy on PubMedQA . the proposed approach provides practical assistance to clinicians and biomedically researchers . |
BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for multi-hop reasoning in biomedical domain are lacking . bioHopR provides benchmarks to evaluate multi-step reasoning in structured biomedic knowledge graphs . |
| Approach: | They propose a benchmark to evaluate multi-hop, multi-answer reasoning in biomedical knowledge graphs. |
| Outcome: | BioHopR evaluates multi-hop reasoning in biomedical knowledge graphs based on the PrimeKG model . it outperforms proprietary models and open-source biomedal models in 1-hop and 2-hop tasks . |
Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Several studies claim that domain-adaptive pretraining improves performance on downstream medical tasks. |
| Approach: | They compare medical LLMs and VLMs against their corresponding base models . they find that medical Lms outperform their base models in 12.1% of cases . |
| Outcome: | The proposed models outperform their base models on medical questions and tasks in 12.1% of cases and reach a tie in 49.8% of cases. |
ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning (2025.emnlp-main)
Copied to clipboard
Yu Sun, Xingyu Qian, Weiwen Xu, Hao Zhang, Chenghao Xiao, Long Li, Deli Zhao, Wenbing Huang, Tingyang Xu, Qifeng Bai, Yu Rong
| Challenge: | Existing medical reasoning datasets are limited in scale and typically rely on incomplete data. |
| Approach: | They propose to use ReasonMed to train medical reasoning models using a multi-agent generation, verification, and refinement pipeline. |
| Outcome: | The largest medical reasoning dataset to date surpasses the prior best sub-10B models by 4.17% and even exceeds LLaMA3.1-70B on PubMedQA by 4.60%. |
Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewards (2025.emnlp-main)
Copied to clipboard
Jaehoon Yun, Jiwoong Sohn, Jungwoo Park, Hyunjae Kim, Xiangru Tang, Daniel Shao, Yong Hoe Koo, Ko Minhyeok, Qingyu Chen, Mark Gerstein, Michael Moor, Jaewoo Kang
| Challenge: | Large language models have shown promise in clinical decision making, but current approaches struggle to localize and correct reasoning errors at specific steps of the reasoning process. |
| Approach: | They propose a process reward modeling framework that leverages retrieval-augmented generation to verify each reasoning step against established medical knowledge bases. |
| Outcome: | The proposed model improves on five medical QA benchmarks and two open-ended diagnostic tasks by 13.50% on MedQA. |