IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for detecting Indirect Prompt Injection (IPI) attacks rely on assumptions about the model's inherent security, which lacks structural constraints on agent behaviors. |
| Approach: | They propose a novel task execution paradigm that models the agents’ task execution process as a traversal over a planned Tool Dependency Graph (TDG). |
| Outcome: | The proposed model reduces unintended tool invocations triggered by injected instructions, enhancing robustness against IPI attacks. |
Similar Papers
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work has embodied LLMs as agents, but external content introduces the risk of indirect prompt injection (IPI) attacks. |
| Approach: | They propose a benchmark to assess the vulnerability of tool-integrated LLM agents to IPI attacks. |
| Outcome: | The proposed benchmark assesses the vulnerability of tool-integrated LLM agents to IPI attacks. |
The Dangers of Indirect Prompt Injection Attacks on LLM-based Autonomous Web Navigation Agents: A Demonstration (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Large Language Model (LLM)-integrated applications are becoming more popular to support, augment, and automate tasks. |
| Approach: | They propose to embed universal adversarial triggers in webpage HTML to hijack agents . they also use a browser-gym agent powered by Llama-3.1 to test their system . |
| Outcome: | The proposed system software is released under the MIT License . |
The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Model (LLM) agents are becoming conversational assistants . indirect prompt injection attacks pose a critical threat to these systems . |
| Approach: | They propose a novel and orthogonal perspective that reframes agent security . they propose 'task shield' that verifies whether each instruction and tool call contributes to user objectives . |
| Outcome: | The proposed defense reduces attack success rates while maintaining high task utility on the AgentDojo benchmark. |
Defending against Indirect Prompt Injection by Instruction Detection (2025.findings-emnlp)
Copied to clipboard
Tongyu Wen, Chenglong Wang, Xiyuan Yang, Haoyu Tang, Yueqi Xie, Lingjuan Lyu, Zhicheng Dou, Fangzhao Wu
| Challenge: | Indirect Prompt Injection attacks can be exploited by LLMs that are embedded with external data. |
| Approach: | They propose a detection-based approach that leverages the behavioral states of LLMs to identify potential IPI attacks. |
| Outcome: | The proposed approach reduces the success rate of attacks to 0.03% on the BIPIA benchmark. |
Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) agents use external tools to interact with environments, but they also introduce security risks, such as indirect prompt injection (IPI) attacks. |
| Approach: | They evaluate eight different defenses and bypass all of them using adaptive attacks. |
| Outcome: | The proposed defenses achieve an attack success rate of over 50% and are robust against adaptive attacks. |
From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injection (2025.naacl-long)
Copied to clipboard
| Challenge: | Toolcalling has changed Large Language Model (LLM) applications by integrating external tools, but it also introduces new security vulnerabilities, particularly in the tool scheduling mechanisms of LLM, which have not been extensively studied. |
| Approach: | They propose a framework that exploits vulnerabilities in Large Language Models through adversarial tool injection to execute privacy theft, launch denial-of-service attacks, and manipulate business competition. |
| Outcome: | The proposed framework exploits vulnerabilities in LLM tool-calling systems through adversarial tool injection. |
Know Thy Enemy: Securing LLMs Against Prompt Injection via Diverse Data Synthesis and Instruction-Level Chain-of-Thought Learning (2026.findings-acl)
Copied to clipboard
Zhiyuan Chang, Mingyang Li, Yuekai Huang, Ziyou Jiang, Xiaojun Jia, Qian Xiong, Junjie Wang, Zhaoyang Li, Qing Wang
| Challenge: | Large language model (LLM)-integrated applications face security vulnerabilities from prompt injection (PI) attacks. |
| Approach: | They propose a model enhancement method that synthesizes diverse training data and employs instruction-level chain-of-thought fine-tuning to enable LLMs to effectively identify and reject malicious instructions regardless of their source or position in the context. |
| Outcome: | The proposed method outperforms baselines in three critical dimensions while maintaining utility performance without degradation. |
VortexPIA: Indirect Prompt Injection Attack against LLMs for Efficient Extraction of User Privacy (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been widely deployed in Conversational AIs . however, the methods proposed in the study rely on a white-box setting . |
| Approach: | They propose an indirect prompt injection attack that induces privacy extraction in LLMs . they use token-efficient data containing false memories to inject LLM data . |
| Outcome: | The proposed method outperforms baselines and achieves state-of-the-art performance. |
PIGuard: Prompt Injection Guardrail via Mitigating Overdefense for Free (2025.acl-long)
Copied to clipboard
| Challenge: | Prompt injection attacks pose a critical threat to large language models, enabling goal hijacking and data leakage. |
| Approach: | They propose a prompt guard model that incorporates a new training strategy to mitigate over-defense for free . PIGuard significantly reduces the bias on trigger words, enabling fine-grained evaluation . |
| Outcome: | The proposed model outperforms the existing model on diverse benchmarks by 30.4%. |
Invisible Prompts, Visible Threats: Malicious Font Injection in External Resources for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly equipped with capabilities of real-time web search and integrated with protocols like the Model Context Protocol (MCP). |
| Approach: | They investigate the vulnerability of Large Language Models to hidden adversarial prompts . they evaluate two critical attack scenarios: malicious content relay and sensitive data leakage . |
| Outcome: | The proposed extension could introduce new security vulnerabilities. |