SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Model (LLM) agents are expanding their action spaces to operate in complex environments. |
| Approach: | They propose a server-side defense plugin that constrains tool acquisition via predictive reasoning regarding future safety risks. |
| Outcome: | Experiments on PowerSeeking Bench, ToolEmu, and AgentHarm show that SafeMCP achieves a safe equilibrium, effectively mitigating risks while preserving agent utility. |
Similar Papers
Securing the Tool Layer: A Threat Taxonomy and Runtime Defense Framework for Model Context Protocol Deployments (2026.acl-industry)
Copied to clipboard
| Challenge: | Model Context Protocol (MCP) is the dominant standard for connecting large language models to external tools, databases, and services. |
| Approach: | They propose a runtime security framework that performs real-time validation of MCP tool calls and responses. |
| Outcome: | The proposed framework reduces attack success rates from 74% to under 9% for tool poisoning and from 47% to under6% for indirect prompt injection via tool responses. |
MCP-Guard: A Multi-Stage Defense-in-Depth Framework for Securing Model Context Protocol in Agentic AI (2026.findings-acl)
Copied to clipboard
Wenpeng Xing, Zhonghao Qi, Yupeng Qin, Yilin Li, Caini Chang, Jiahui Yu, Changting Lin, Zhenzhen Xie, Meng Han
| Challenge: | Large Language Models (LLMs) are vulnerable to jailbreak, authors say . authors propose a robust, layered defense architecture designed for LLM–tool interactions . |
| Approach: | They propose a robust, layered defense architecture designed for LLM–tool interactions . they propose XCP-Guard, which employs a three-stage detection pipeline . |
| Outcome: | The proposed model achieves 96.01% accuracy in identifying adversarial prompts . the model is based on a three-stage detection pipeline that balances efficiency with accuracy . |
Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing alignment methods focus on reactive feedback, where immediate human perception is leveraged to judge sampled model responses as preference data for post-training. |
| Approach: | They propose a proof-of-concept framework that projects how model-generated advice could propagate through societal systems on a macroscopic scale over time, enabling more robust alignment. |
| Outcome: | The proposed framework achieves 20% improvement on existing safety benchmarks and an average win rate exceeding 70% against strong baselines. |
ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback (2026.findings-acl)
Copied to clipboard
| Challenge: | Unlike chatbots, autonomous agents act directly on external environments, making tool invocation safety critical for reliable deployment. |
| Approach: | They develop a benchmark for step-level tool invocation safety detection in LLM agents and a guardrail model that proactively detects unsafe tool invoking actions before execution using multi-task reinforcement learning. |
| Outcome: | The proposed model reduces harmful tool invocations of ReAct-style agents by 65% on average and improves benign task completion by 10% under prompt injection attacks. |
ToolCPT: Improving Tool Utilization in LLM Agents via Continuous Pre-training (2026.findings-acl)
Copied to clipboard
| Challenge: | Current approaches to enhancing tool use for LLM-based agents focus on post-training fine-tuning or test-time context extension. |
| Approach: | They propose to enhance tool knowledge for LLM-based agents during continuous pre-training . they curate 5.1 million code artifacts from large-scale, high-quality code repositories . |
| Outcome: | The proposed model outperforms existing methods on out-of-distribution tools on multiple benchmarks. |
SafeAgent: Safeguarding LLM Agents via an Automated Risk Simulator (2026.acl-long)
Copied to clipboard
Xueyang Zhou, Weidong Wang, Lin Lu, Jiawen Shi, Guiyao Tie, Xu Yongtian, Lixing Chen, Pan Zhou, Neil Zhenqiang Gong, Lichao Sun
| Challenge: | SafeAgent improves agent safety through fully automated synthetic data generation. |
| Approach: | They propose a framework that improves agent safety through fully automated synthetic data generation. |
| Outcome: | The proposed framework outperforms closed-source models on two safety benchmarks and one real-world task. |
SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that fine-tuning can erode safety alignment, causing LLMs to respond to harmful or unethical prompts. |
| Approach: | They propose a lightweight framework that restores safety while maintaining downstream performance. |
| Outcome: | The proposed framework reduces harmful outputs compared to other defenses, with negligible impact on utility. |
MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP Tools (2026.acl-long)
Copied to clipboard
WenHao Wang, Peizhi Niu, Zhao Xu, Zhaoyu Chen, Jian Du, Yaxin Du, Xianghe Pang, Keduan Huang, Yanfeng Wang, Qiang Yan, Siheng Chen
| Challenge: | Existing research on Large Language Models (LLMs) relies on few servers and lacks training support. |
| Approach: | They propose a web-agent-driven pipeline for large-scale server discovery, data synthesis, and model training that collects and filters data from 1166 servers and 11536 tools. |
| Outcome: | Empirical evidence shows that MCP-Flow generates higher quality instruction-function call pairs and higher agentic task performance than previous work. |
MCIP: Protecting MCP Safety via Model Contextual Integrity Protocol (2025.emnlp-main)
Copied to clipboard
| Challenge: | Model Context Protocol (MCP) introduces an easy-to-use ecosystem for users and developers, but it also brings underexplored safety risks. |
| Approach: | They propose a framework that addresses the missing safety mechanisms in MCP and a taxonomy that captures diverse range of unsafe behaviors observed in MMP scenarios. |
| Outcome: | The proposed framework improves safety performance on state-of-the-art LLMs by capturing unsafe behaviors and analyzing the results. |
Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safety (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are vulnerable to jailbreak attacks that exploit weaknesses in traditional safety alignment. |
| Approach: | They propose a framework that trains models to engage in explicit safe reasoning before response . they propose RATIONAL, which allows models to reject harmful prompts while providing meaningful and context-aware responses. |
| Outcome: | The proposed framework fine-tunes models to reason about query intent, ethics, and potential harm. |