Efficient but Vulnerable: Benchmarking and Defending LLM Batch Prompting Attack (2025.findings-acl)
Copied to clipboard
| Challenge: | a recent study reveals a significant security vulnerability in batch prompting . malicious users can inject attack instructions into a batch, leading to unwanted interference . |
| Approach: | They construct a batch prompting benchmark and test it against other LLMs to find out if batch prompts are vulnerable. |
| Outcome: | The proposed approach achieves 95% accuracy in detecting attacks. |
Similar Papers
Defense Against Prompt Injection Attack by Leveraging Attack Techniques (2025.acl-long)
Copied to clipboard
| Challenge: | Recent attacks leverage LLMs’ instruction-following abilities and their inabilities to distinguish instructions injected in the data content. |
| Approach: | They invert the intention of prompt injection methods to develop novel defense methods based on previous training-free attack methods by repeating the attack process with the original input instruction rather than the injected instruction. |
| Outcome: | The proposed methods outperform existing defense approaches, achieving state-of-the-art results. |
Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction (2026.findings-acl)
Copied to clipboard
Yulin Chen, Haoran Li, Yuan Sui, Yue Liu, Yufei He, Xiaoling Bai, Chi Fei, Li Yabo, Haozhe Ma, Yangqiu Song, Bryan Hooi
| Challenge: | Prompt injection attacks manipulate large language models (LLMs) by misleading them to deviate from the original input instructions and execute maliciously injected instructions. |
| Approach: | They propose a prompt injection defense method that suppresses the model's instruction-following tendencies rather than suppressing them. |
| Outcome: | The proposed method outperforms prompt-engineering-based approaches and fine-tuning methods and reduces the ASR to nearly 0% in some scenarios. |
Rethinking Assessments of Prompt Injection Attacks (2026.findings-acl)
Copied to clipboard
| Challenge: | Prompt injection attacks are recognized as one of the primary risks faced by LLM-integrated applications in recent years. |
| Approach: | They evaluate prompt injection attacks on LLM-integrated applications across 37 target tasks, 185 injected tasks, 21 attack instructions, and 143,745 queries. |
| Outcome: | The proposed framework provides a solid foundation for assessing vulnerabilities in LLM-integrated applications and evaluating the efficacy of defensive strategies. |
Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional proficiency in instruction-following, making them increasingly integral to various applications. |
| Approach: | They establish a benchmark to evaluate the robustness of instruction-following LLMs against prompt injection attacks, assessing their ability to discern which instructions to follow and which to disregard. |
| Outcome: | The proposed model is overly sensitive to prompt injection attacks, focusing on the latter part of the prompt without fully understanding the context. |
Batch Prompting: Efficient Inference with Large Language Model APIs (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Performing inference on large volumes of samples can be computationally and financially costly. |
| Approach: | They propose a prompting approach that enables large language models to run inference in batches instead of one sample at a time. |
| Outcome: | The proposed prompting reduces both token and time costs while retaining downstream performance. |
Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on poisoning during supervised fine-tuning, RLHF, or inference-time time optimization. |
| Approach: | They propose a simple fake reward attack that requires no access to the reward model and significantly increases vulnerability. |
| Outcome: | The proposed attack reduces the fake reward ASR from 0.23 to 0.07 without degrading utility. |
The Threat of PROMPTS in Large Language Models: A System and User Prompt Perspective (2025.findings-acl)
Copied to clipboard
| Challenge: | Prompts are essential for guiding model output and influencing content generation. |
| Approach: | They propose to attack models with prompt leakage and prompt jailbreak attacks . they summarize the experimental setups of these methods and explore the relationship between prompt threats and prompt injection attacks. |
| Outcome: | The proposed methods summarize the experimental setups and examine the relationship between prompt threats and prompt injection attacks. |
LLMs Caught in the Crossfire: Malware Requests and Jailbreak Challenges (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have a high vulnerability to jailbreak attacks that leverage crafted prompts to generate malicious outputs. |
| Approach: | They propose to use large language models to test their security against jailbreak attacks that leverage crafted prompts to generate malicious outputs. |
| Outcome: | The proposed model is based on 320 manually crafted malicious code generation requirements, covering 11 jailbreak methods and 29 code functionality categories. |
The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | cipher decoding, riddles, code execution embedded into model prompts bypass safety safeguards of large language models (LLMs) . |
| Approach: | They introduce a novel class of adversarial jailbreak adversarials on large language models, termed Task-in-Prompt (TIP) attacks. |
| Outcome: | The proposed techniques circumvent safeguards in six state-of-the-art language models, including GPT-4o and LLaMA 3.2, and consistently generate restricted content . |
Conjunctive Prompt Attacks in Multi-Agent LLM Systems (2026.acl-long)
Copied to clipboard
| Challenge: | Existing defenses do not reliably stop the attack because no single component appears malicious in isolation. |
| Approach: | They study conjunctive prompt attacks where trigger key and adversarial template appear benign alone but activate harmful behavior when routing brings them together. |
| Outcome: | The proposed model significantly improves performance over baselines while keeping false activations low. |