Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show superior cross-model transferability . Prompt injection attacks are among the most critical threats . |
| Approach: | They propose an activations-guided prompt injection attack framework to address the impracticality of existing white-box/gray-box methods and the poor transferability of black-box approaches. |
| Outcome: | The proposed framework achieves 49.6% success rate and 34.6% improvement over human-crafted prompts on five mainstream LLMs. |
Similar Papers
Exploiting Prompt-induced Confidence for Black-Box Attacks on LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are vulnerable to adversarial attacks even in strict black-box settings with only hard-label feedback. |
| Approach: | They propose a black-box framework that leverages prompt-induced confidence as an auxiliary signal to guide attacks. |
| Outcome: | The proposed framework improves the attack success rate and query efficiency while maintaining semantic fidelity. |
Rethinking Assessments of Prompt Injection Attacks (2026.findings-acl)
Copied to clipboard
| Challenge: | Prompt injection attacks are recognized as one of the primary risks faced by LLM-integrated applications in recent years. |
| Approach: | They evaluate prompt injection attacks on LLM-integrated applications across 37 target tasks, 185 injected tasks, 21 attack instructions, and 143,745 queries. |
| Outcome: | The proposed framework provides a solid foundation for assessing vulnerabilities in LLM-integrated applications and evaluating the efficacy of defensive strategies. |
Know Thy Enemy: Securing LLMs Against Prompt Injection via Diverse Data Synthesis and Instruction-Level Chain-of-Thought Learning (2026.findings-acl)
Copied to clipboard
Zhiyuan Chang, Mingyang Li, Yuekai Huang, Ziyou Jiang, Xiaojun Jia, Qian Xiong, Junjie Wang, Zhaoyang Li, Qing Wang
| Challenge: | Large language model (LLM)-integrated applications face security vulnerabilities from prompt injection (PI) attacks. |
| Approach: | They propose a model enhancement method that synthesizes diverse training data and employs instruction-level chain-of-thought fine-tuning to enable LLMs to effectively identify and reject malicious instructions regardless of their source or position in the context. |
| Outcome: | The proposed method outperforms baselines in three critical dimensions while maintaining utility performance without degradation. |
Defense Against Prompt Injection Attack by Leveraging Attack Techniques (2025.acl-long)
Copied to clipboard
| Challenge: | Recent attacks leverage LLMs’ instruction-following abilities and their inabilities to distinguish instructions injected in the data content. |
| Approach: | They invert the intention of prompt injection methods to develop novel defense methods based on previous training-free attack methods by repeating the attack process with the original input instruction rather than the injected instruction. |
| Outcome: | The proposed methods outperform existing defense approaches, achieving state-of-the-art results. |
Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction (2026.findings-acl)
Copied to clipboard
Yulin Chen, Haoran Li, Yuan Sui, Yue Liu, Yufei He, Xiaoling Bai, Chi Fei, Li Yabo, Haozhe Ma, Yangqiu Song, Bryan Hooi
| Challenge: | Prompt injection attacks manipulate large language models (LLMs) by misleading them to deviate from the original input instructions and execute maliciously injected instructions. |
| Approach: | They propose a prompt injection defense method that suppresses the model's instruction-following tendencies rather than suppressing them. |
| Outcome: | The proposed method outperforms prompt-engineering-based approaches and fine-tuning methods and reduces the ASR to nearly 0% in some scenarios. |
TopicAttack: An Indirect Prompt Injection Attack via Topic Transition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent attacks have demonstrated potential, but their abrupt instruction injection often undermines their effectiveness. |
| Approach: | They propose a method that prompts the LLM to generate a fabricated conversational transition prompt that gradually shifts the topic toward the injected instruction. |
| Outcome: | The proposed method achieves state-of-the-art performance with an attack success rate (ASR) over 90% in most cases, even when various defense methods are applied. |
Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs Through a Global Prompt Hacking Competition (2023.emnlp-main)
Copied to clipboard
Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis-François Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Kost, Christopher Carnahan, Jordan Boyd-Graber
| Challenge: | Large Language Models are increasingly being deployed in interactive contexts that involve direct user engagement. |
| Approach: | They run a global prompt hacking competition to encourage research on prompt hacks . they elicit 600K+ adversarial prompts against three state-of-the-art LLMs based on a dataset . |
| Outcome: | The results of the competition show that current LLMs can be manipulated via prompt hacking . the competition elicits 600K+ adversarial prompts against three state-of-the-art LLM models . |
Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional proficiency in instruction-following, making them increasingly integral to various applications. |
| Approach: | They establish a benchmark to evaluate the robustness of instruction-following LLMs against prompt injection attacks, assessing their ability to discern which instructions to follow and which to disregard. |
| Outcome: | The proposed model is overly sensitive to prompt injection attacks, focusing on the latter part of the prompt without fully understanding the context. |
Conjunctive Prompt Attacks in Multi-Agent LLM Systems (2026.acl-long)
Copied to clipboard
| Challenge: | Existing defenses do not reliably stop the attack because no single component appears malicious in isolation. |
| Approach: | They study conjunctive prompt attacks where trigger key and adversarial template appear benign alone but activate harmful behavior when routing brings them together. |
| Outcome: | The proposed model significantly improves performance over baselines while keeping false activations low. |
LinkPrompt: Natural and Universal Adversarial Attacks on Prompt-based Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Prompt-based learning is a new language model training paradigm that adapts Pre-trained Language Models (PLMs) to downstream tasks. |
| Approach: | They propose a prompt-based learning paradigm that adapts Pre-trained Language Models to downstream tasks . they use a gradient-based beam search algorithm to generate adversarial triggers . |
| Outcome: | The proposed model improves performance on various natural language processing tasks by optimizing the prompt template. |