MeNTi: Bridging Medical Calculator and LLM Agent with Nested Tool Calling (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been widely used in medicine but are limited in their ability to fully address the complexities of the real world. |
| Approach: | They propose a universal agent architecture for Large Language Models that integrates a specialized medical toolkit and employs meta-tool and nested calling mechanisms to enhance LLM tool utilization. |
| Outcome: | The proposed framework improves the accuracy and performance of medical calculators in complex medical scenarios. |
Similar Papers
MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integration (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on static single-step calculations with explicit instructions. |
| Approach: | They propose a benchmark for evaluating medical calculators in realistic scenarios . they use 118 scenario tasks across 4 clinical domains to evaluate medical calculator performance . |
| Outcome: | The first benchmark for evaluating medical calculators in realistic scenarios is released . it features 118 scenario tasks across 4 clinical domains and is based on a model context protocol integration. |
A Survey of LLM-based Agents in Medicine: How far are we from Baymax? (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are transforming healthcare through their ability to understand and assist with medical tasks. |
| Approach: | They analyze system profiles, clinical planning, medical reasoning frameworks, and external capacity enhancement. |
| Outcome: | The findings highlight the future directions in medical reasoning, physical system integration, and training simulations. |
From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations (2025.emnlp-main)
Copied to clipboard
Benlu Wang, Iris Xia, Yifan Zhang, Junda Wang, null Feiyun Ouyang, Shuo Han, Arman Cohan, Hong Yu, Zonghai Yao
| Challenge: | Existing benchmarks assess only the final answer with a wide numerical tolerance, overlooking systematic reasoning failures and potentially causing serious clinical misjudgments. |
| Approach: | They propose a new step-by-step evaluation pipeline that assesses formula selection, entity extraction, and arithmetic computation. |
| Outcome: | The proposed method improves the accuracy of large language models on medical benchmarks from 16.35% to 53.19%. |
ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filtering (2026.acl-long)
Copied to clipboard
| Challenge: | Large language model (LLM) agents often face strict input context limits, preventing efficient consideration of large toolsets. |
| Approach: | They propose a tool that allows LLMs to merge tools with auto-correction and toolScopeRetriever to rank and select only the most relevant tools for each query. |
| Outcome: | Evaluations on three state-of-the-art LLMs and three open-source tool-use benchmarks show gains of 8.38% to 38.6% in tool selection accuracy. |
LLM Agents Making Agent Tools (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can perform multi-step tasks by dynamically utilising external software components. |
| Approach: | They propose an agentic framework that autonomously transforms papers with code into LLM-compatible tools. |
| Outcome: | The proposed framework outperforms current state-of-the-art software engineering agents in 80% of tasks and is openly available on GitHub. |
MMedAgent: Learning to Use Medical Tools with Multi-modal Agent (2024.findings-emnlp)
Copied to clipboard
Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, Yixin Wang
| Challenge: | Multi-modal Large Language Models (MLLMs) exhibit limited generality and often fall short when compared to specialized models. |
| Approach: | They propose a multi-modal medical agent that picks the most suitable medical tools based on user inputs. |
| Outcome: | The proposed agent performs better than open-source models and the closed-source model, GPT-4o. |
Meta-Tool: Unleash Open-World Function Calling Capabilities of General-Purpose Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models struggle with addressing diverse user inquiries in open-world tasks. |
| Approach: | They propose a plug-and-play tool retrieval system for LLMs to access external tool library and use retrieved tools to solve user's problem. |
| Outcome: | The proposed model improves on a finetuned version of LLaMA-3.1 and 2,800 dialogues and 7,361 tools spanning ten distinct test categories. |
ACEBench: A Comprehensive Evaluation of LLM Tool Usage (2025.findings-emnlp)
Copied to clipboard
Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Yuefeng Huang, Xiangcheng Liu, Wang Xinzhi, Wu Liu
| Challenge: | Existing benchmarks for evaluating LLMs’ tool usage face several limitations: limited evaluation scenarios, lacking assessments in real multi-turn dialogue contexts; narrow evaluation dimensions, with insufficient detailed assessments of how LLM use tools; and reliance on LLM or real API executions for evaluation, which introduces significant overhead. |
| Approach: | ACEBench is a benchmark for evaluating tool usage in Large Language Models . it categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. |
| Outcome: | ACEBench categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. |
Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are generalist agents capable of operating within complex environments. |
| Approach: | They propose a class of tools that can serve as a middleware layer shielding LLMs from environmental complexity. |
| Outcome: | The proposed tool can shield the LLM from environmental complexity in two representative complex environments. |
BioTool: A Comprehensive Tool-Calling Dataset for Enhancing Biomedical Capabilities of Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | despite the success of large language models, their performance in highly specialized domains remains unsatisfactory. |
| Approach: | They propose a biomedical tool-calling dataset designed for fine-tuning LLMs . the dataset contains 34 frequently used tools from the NCBI, Ensembl, and UniProt databases . |
| Outcome: | The proposed dataset outperforms commercial LLMs on biomedical domains. |