Papers with AgentHarm
Agent vs. Agent: Automated Data Generation and Red-Teaming for Custom Agentic Workflows (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing red-teaming frameworks like AgentHarm use static prompts and hardcoded toolsets . |
| Approach: | They propose a red-teaming framework that generates adversarial tasks and evaluation functions tailored to arbitrary toolsets and uses iterative prompt refinement with self-reflection to develop more effective attacks. |
| Outcome: | The proposed approach achieves 162% increase in attack success rate on o4-mini and 86% success on gemini 2.5 Pro. |
SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Model (LLM) agents are expanding their action spaces to operate in complex environments. |
| Approach: | They propose a server-side defense plugin that constrains tool acquisition via predictive reasoning regarding future safety risks. |
| Outcome: | Experiments on PowerSeeking Bench, ToolEmu, and AgentHarm show that SafeMCP achieves a safe equilibrium, effectively mitigating risks while preserving agent utility. |