MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing factuality evaluation pipelines are poor matches for medical domains . existing methods are limited to objective, entity-centric, formulaic texts . |
| Approach: | They propose a pipeline to decompose medical answers into condition-aware valid facts . they use a decomposition-then-verify approach to evaluate generated text . |
| Outcome: | The proposed method extracts up to three times as many valid facts as existing methods . the resulting factuality score substantially varies by decomposition method, corpus, and used backbone LLM . |
Similar Papers
LEAF: Learning and Evaluation Augmented by Fact-Checking to Improve Factualness in Large Language Models (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) struggle with factual accuracy in knowledge-intensive domains like healthcare. |
| Approach: | They propose a framework for improving LLM factuality in medical question answering . RAFE, Fact-Check-then-RAG and Learning from Fact Check are components . |
| Outcome: | Experimental results show that LEAF outperforms Factcheck-GPT in detecting inaccuracies and corrects errors without labeling . the framework provides a scalable solution for industrial applications requiring high factuality scores. |
imapScore: Medical Fact Evaluation Made Easy (2024.findings-acl)
Copied to clipboard
| Challenge: | Automated evaluation of natural language generation tasks fails to focus on medical QA because of the diversity in medical terminology. |
| Approach: | They propose a new data structure, imap, to capture key information in questions and answers. |
| Outcome: | The proposed model outperforms state-of-the-art metrics in correlation with human scores. |
Beyond Pointwise Scores: Decomposed Criteria-Based Evaluation of LLM Responses (2025.emnlp-industry)
Copied to clipboard
| Challenge: | DeCE is model-agnostic and domain-general, requiring no predefined taxonomies or handcrafted rubrics. |
| Approach: | They propose a decomposed LLM evaluation framework that separates accuracy and recall from accuracy and relevance. |
| Outcome: | The proposed framework achieves stronger correlation with expert judgments than traditional metrics and pointwise LLM scoring. |
From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations (2025.emnlp-main)
Copied to clipboard
Benlu Wang, Iris Xia, Yifan Zhang, Junda Wang, null Feiyun Ouyang, Shuo Han, Arman Cohan, Hong Yu, Zonghai Yao
| Challenge: | Existing benchmarks assess only the final answer with a wide numerical tolerance, overlooking systematic reasoning failures and potentially causing serious clinical misjudgments. |
| Approach: | They propose a new step-by-step evaluation pipeline that assesses formula selection, entity extraction, and arithmetic computation. |
| Outcome: | The proposed method improves the accuracy of large language models on medical benchmarks from 16.35% to 53.19%. |
BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains (2024.findings-acl)
Copied to clipboard
Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, Richard Dufour
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable versatility in recent years, offering potential applications across specialized domains such as healthcare and medicine. |
| Approach: | They propose an open-source LLM tailored for the biomedical domain that utilizes Mistral as its foundation model and pre-trained on PubMed Central. |
| Outcome: | The proposed model outperforms existing models on a benchmark comprising 10 established medical question-answering tasks in English and is competitive with proprietary models. |
Beyond Memorization: A Rigorous Evaluation Framework for Medical Knowledge Editing (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing knowledge editing methods show promising results on general-domain benchmarks, but their effectiveness in the medical domain remains largely unexplored. |
| Approach: | They propose a framework to evaluate medical knowledge editing using model-generated rationales as editing targets. |
| Outcome: | The proposed method improves editing efficacy and generalization in medical models without full retraining. |
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation (2025.findings-emnlp)
Copied to clipboard
Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Current medical benchmarks have limitations in question design, data sources and evaluation methods. |
| Approach: | They propose a new benchmark covering five core medical areas . it includes 2,996 questions created from real-world electronic health records . |
| Outcome: | The proposed model covers five core medical areas and includes 2,996 questions created from real-world electronic health records and expert-designed clinical scenarios. |
FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing factuality evaluation models are not robust, especially with respect to entity and relation errors in new domains. |
| Approach: | They propose a new approach to factuality evaluation that is generalizable across domains . they propose entities-specific facts, facts extracted from external knowledge bases and facts constructed compositionally through knowledge base walks. |
| Outcome: | The proposed model achieves state-of-the-art on two in-domain news summarization benchmarks and on three out-of domain scientific literature datasets. |
Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Several studies claim that domain-adaptive pretraining improves performance on downstream medical tasks. |
| Approach: | They compare medical LLMs and VLMs against their corresponding base models . they find that medical Lms outperform their base models in 12.1% of cases . |
| Outcome: | The proposed models outperform their base models on medical questions and tasks in 12.1% of cases and reach a tie in 49.8% of cases. |
VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work focuses on accuracy and precision, but factuality evaluation is difficult due to inter-sentence dependencies. |
| Approach: | They introduce a factuality evaluation framework to enhance fact extraction . they also introduce 'factRBench' that evaluates both precision and recall . |
| Outcome: | The proposed framework enhances fact extraction by identifying incomplete and missing facts . it also evaluates precision and recall in long-form models, whereas prior work focuses on precision. |