CReSE: Benchmark Data and Automatic Evaluation Framework for Recommending Eligibility Criteria from Clinical Trial Information (2024.findings-eacl)
Copied to clipboard
| Challenge: | Eligibility criteria (EC) are defined as a set of conditions an individual must meet to participate in a clinical trial. |
| Approach: | They propose to recommend EC based on clinical trial information, including trial titles, and introduce an automatic evaluation framework to assess clinical validity of the EC recommendation model. |
| Outcome: | The proposed model outperforms existing language models pre-trained on the biomedical domain in EC clustering. |
Similar Papers
EC-RAFT: Automated Generation of Clinical Trial Eligibility Criteria through Retrieval-Augmented Fine-Tuning (2025.findings-acl)
Copied to clipboard
| Challenge: | Eligibility criteria (EC) are critical components of clinical trial design, specifying parameters for participant inclusion and exclusion. |
| Approach: | They propose a method that utilizes Retrieval-Augmented Fine-Tuning to generate structured and cohesive EC directly from clinical trial titles and descriptions. |
| Outcome: | The proposed method outperforms Llama-3.1-8B-Instruct and Llm-as-a-Judge models in BERTScore and EC score. |
Dataset and Enhanced Model for Eligibility Criteria-to-SQL Semantic Parsing (2020.lrec-1)
Copied to clipboard
| Challenge: | Clinical trials require that patients meet eligibility criteria to ensure safety and effectiveness of studies. |
| Approach: | They propose a dataset that includes the first-of-its-kind eligibility-criteria corpus and queries for criteria-to-sql . they propose 'neuro semantic parser' which can translate eligibility criteria to executable SQL queries . |
| Outcome: | The proposed parser outperforms existing state-of-the-art general-purpose models while highlighting the challenges presented by the new dataset. |
AutoTrial: Prompting Language Models for Clinical Trial Design (2023.emnlp-main)
Copied to clipboard
| Challenge: | Generative large language models (LLMs) are a popular tool for creating coherent and human-like documents for clinical trials. |
| Approach: | They propose to generate clinical eligibility criteria using language models by a hybrid of discrete and neural prompting and scalable knowledge incorporation via in-context learning. |
| Outcome: | The proposed method generates high-quality criteria texts fluent and coherent with high accuracy against the GPT-3.5 baselines. |
ProMedical: Hierarchical Fine-Grained Criteria Modeling for Medical LLM Alignment via Explicit Injection (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are difficult to align with high-stakes medical standards due to dissonance between coarse-grained preference signals and complex protocols. |
| Approach: | They propose a framework that aligns Large Language Models with medical standards . they use a dataset generated via a human-in-the-loop pipeline to augment medical instructions . |
| Outcome: | The proposed framework disentangles safety constraints from general proficiency, enabling precise guidance during reinforcement learning. |
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation (2025.findings-emnlp)
Copied to clipboard
Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Current medical benchmarks have limitations in question design, data sources and evaluation methods. |
| Approach: | They propose a new benchmark covering five core medical areas . it includes 2,996 questions created from real-world electronic health records . |
| Outcome: | The proposed model covers five core medical areas and includes 2,996 questions created from real-world electronic health records and expert-designed clinical scenarios. |
Exploring the Generalization of Cancer Clinical Trial Eligibility Classifiers across Diseases (2024.lrec-main)
Copied to clipboard
| Challenge: | Clinical trials are pivotal in medical research, and NLP can enhance their success with application in recruitment. |
| Approach: | They examine the generalizability of eligibility classification across clinical trials . they use an extensive cancer dataset to examine how well models can handle criteria . |
| Outcome: | The proposed model can handle criteria commonly found in non-cancer trials, but struggle with criteria disproportionately prevalent in cancer trials. |
MedQA-CS: Objective Structured Clinical Examination (OSCE)-Style Benchmark for Evaluating LLM Clinical Skills (2026.eacl-long)
Copied to clipboard
Zonghai Yao, Zihao Zhang, Chaolong Tang, Xingyu Bian, Youxia Zhao, Zhichao Yang, Junda Wang, Huixue Zhou, Won Seok Jang, Feiyun Ouyang, Hong Yu
| Challenge: | Current clinical LLM benchmarks fail to evaluate advanced clinical skills in AI and large language models (LLMs). |
| Approach: | They propose a framework to evaluate large language models (LLMs) using two instruction-following tasks designed to reflect real clinical scenarios. |
| Outcome: | The proposed framework evaluates LLMs through two instruction-following tasks designed to reflect real clinical scenarios. |
AutoMedEval: Harnessing Language Models for Automatic Medical Capability Evaluation (2025.acl-long)
Copied to clipboard
Xiechi Zhang, Zetian Ouyang, Linlin Wang, Gerard De Melo, Zhu Cao, Xiaoling Wang, Ya Zhang, Yanfeng Wang, Liang He
| Challenge: | Existing evaluation methods based on large language models (LLMs) are expensive and lack expertise due to limitations in human expertise. |
| Approach: | They propose an open-source automatic evaluation model with 13B parameters specifically engineered to measure the question-answering proficiency of medical LLMs. |
| Outcome: | The proposed model surpasses baselines in terms of correlation with human judgments. |
Building Trust in Clinical LLMs: Bias Analysis and Dataset Transparency (2025.emnlp-main)
Copied to clipboard
Svetlana Maslenkova, Clement Christophe, Marco AF Pimentel, Tathagata Raha, Muhammad Umar Salman, Ahmed Al Mahrooqi, Avani Gupta, Shadab Khan, Ronnie Rajan, Praveenkumar Kanithi
| Challenge: | Current dataset curation and bias assessment practices lack transparency . current approaches lack a thorough understanding of how data characteristics influence model behavior . |
| Approach: | They propose a comprehensive bias evaluation framework that integrates general benchmarks with a healthcare-specific methodology to probe for biases in a sensitive healthcare context. |
| Outcome: | The proposed approach to bias evaluation leverages established benchmarks and a healthcare-specific methodology. |
CliniBench: A Clinical Outcome Prediction Benchmark for Generative and Encoder-Based Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | generative large language models are being investigated for complex medical tasks, but their effectiveness in real-world clinical applications remains underexplored. |
| Approach: | They propose to compare encoder-based classifiers and generative LLMs for discharge diagnosis prediction from admission notes in a MIMIC-IV dataset. |
| Outcome: | The proposed benchmark compares encoder-based classifiers and generative LLMs for discharge diagnosis prediction from admission notes in the MIMIC-IV dataset. |