ACE-M3: Automatic Capability Evaluator for Multimodal Medical Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing metrics for multimodal large language models only focus on token overlap and may not align with human judgment. |
| Approach: | They propose an open-source model that assesses the question answering abilities of multimodal large language models. |
| Outcome: | Experiments show that the ACE-M3 model performs better than existing models and is more reliable than existing metrics. |
Similar Papers
MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria (2025.naacl-long)
Copied to clipboard
Wentao Ge, Shunian Chen, Hardy Chen, Nuo Chen, Junying Chen, Zhihong Chen, Wenya Xie, Shuo Yan, ChenghaoZhu ChenghaoZhu, Ziyue Lin, Dingjie Song, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jianquan Li, Xiang Wan, Benyou Wang
| Challenge: | Existing evaluation methodologies for multimodal large language models are limited in evaluating objective queries without considering real-world user experiences. |
| Approach: | They propose to evaluate multimodal large language models with per-sample criteria using potent MLLM as the judge. |
| Outcome: | The proposed evaluation paradigm shows that it can be used to evaluate multimodal large language models with per-sample criteria. |
AutoMedEval: Harnessing Language Models for Automatic Medical Capability Evaluation (2025.acl-long)
Copied to clipboard
Xiechi Zhang, Zetian Ouyang, Linlin Wang, Gerard De Melo, Zhu Cao, Xiaoling Wang, Ya Zhang, Yanfeng Wang, Liang He
| Challenge: | Existing evaluation methods based on large language models (LLMs) are expensive and lack expertise due to limitations in human expertise. |
| Approach: | They propose an open-source automatic evaluation model with 13B parameters specifically engineered to measure the question-answering proficiency of medical LLMs. |
| Outcome: | The proposed model surpasses baselines in terms of correlation with human judgments. |
Asclepius: A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Models (2025.acl-long)
Copied to clipboard
Jie Liu, Wenxuan Wang, Su Yihang, Jingyuan Huang, Yudi Zhang, Cheng-Yi Li, Wenting Chen, Xiaohan Xing, Kao-Jung Chang, Linlin Shen, Michael R. Lyu
| Challenge: | Medical Multi-Modal Large Language Models (Med-MLLMs) are a promising new form of artificial general intelligence due to their ability to tackle complex tasks. |
| Approach: | They propose a new benchmark that comprehensively assesses medical multi-modal large language models in terms of distinct medical specialties and different diagnostic capacities. |
| Outcome: | The proposed model covers 15 medical specialties and different diagnostic capacities, and excludes overlap with existing VQA dataset. |
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | MultiModal Large Language Models (MM-LLMs) have undergone significant advances in the past year . traditional MM models incur substantial computational costs, especially when trained from scratch . |
| Approach: | They propose a taxonomy encompassing 126 MM-LLMs and summarize key training recipes to enhance their potency. |
| Outcome: | The proposed models preserve the reasoning and decision-making capabilities of LLMs and empower diverse range of MM tasks. |
AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Evaluating multimodal large language models (MLLMs) is becoming increasingly expensive as benchmarks grow in scale and cross-modality complexity. |
| Approach: | They propose an adaptive evaluation framework for efficient benchmarking that treats evaluation as an interview-like process by keeping a hypothesized ability structure of the evaluated model and actively selecting the informative questions. |
| Outcome: | Experiments on four representative multimodal benchmarks show that **A2-Judger significantly improves sample efficiency while maintaining reliable evaluation results. |
M3Retrieve: Benchmarking Multimodal Retrieval for Medicine (2025.emnlp-main)
Copied to clipboard
| Challenge: | Strong retrieval models are increasingly important in knowledge-intensive domains. |
| Approach: | They propose a benchmark to evaluate multimodal retrieval models in medical settings . they examine 1.2 million text documents and 164K multimodal queries . |
| Outcome: | The proposed model spans 5 domains,16 medical fields, and 4 distinct tasks with over 1.2 Million text documents and 164K multimodal queries. |
MMedAgent: Learning to Use Medical Tools with Multi-modal Agent (2024.findings-emnlp)
Copied to clipboard
Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, Yixin Wang
| Challenge: | Multi-modal Large Language Models (MLLMs) exhibit limited generality and often fall short when compared to specialized models. |
| Approach: | They propose a multi-modal medical agent that picks the most suitable medical tools based on user inputs. |
| Outcome: | The proposed agent performs better than open-source models and the closed-source model, GPT-4o. |
MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models (2026.findings-acl)
Copied to clipboard
Boqi Chen, Xudong Liu, Jiachuan Peng, Marianne Frey-Marti, Kyle Lam, Bang Zheng, Lin Li, Jianing Qiu
| Challenge: | Existing benchmarks for multimodal large language models do not capture real-world clinical complexity. |
| Approach: | They evaluate multilingual, multimodal multimodal models of clinical cases with up to 7 distinct visual clinical evidence types per case. |
| Outcome: | The proposed model outperforms human models on differential diagnosis (DDx) generation and final diagnosis (FDx) selection. |
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation (2025.findings-emnlp)
Copied to clipboard
Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Current medical benchmarks have limitations in question design, data sources and evaluation methods. |
| Approach: | They propose a new benchmark covering five core medical areas . it includes 2,996 questions created from real-world electronic health records . |
| Outcome: | The proposed model covers five core medical areas and includes 2,996 questions created from real-world electronic health records and expert-designed clinical scenarios. |
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge (2026.acl-long)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) are increasingly used as automatic judges . however, their reliability and vulnerabilities to biases remain underexplored . |
| Approach: | They propose a benchmark to evaluate MLLMs that fail to integrate visual cues . they also introduce a test to evaluate the reliability of MLMLs based on a set of asymmetric evaluation tendencies. |
| Outcome: | Experiments on 26 state-of-the-art MLLMs reveal modality neglect and asymmetric evaluation tendencies . a standardized model with a benchmark enables a fine-grained diagnosis of nine bias types . |