Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Text-to-image (T2I) models can be used to generate harmful content such as sexually explicit, unfaithful, and misleading or Not-Safe-for-Work (NSFW) images. |
| Approach: | They propose a more practical and universal attack that does not require the presence of a target model. |
| Outcome: | The proposed attack bypasses both text and image safety checkers while preserving high semantic alignment with the target prompt. |
Similar Papers
Multimodal Pragmatic Jailbreak on Text-to-image Models (2025.acl-long)
Copied to clipboard
Tong Liu, Zhixin Lai, Jiawen Wang, Gengyuan Zhang, Shuo Chen, Philip Torr, Vera Demberg, Volker Tresp, Jindong Gu
| Challenge: | Existing jailbreaks for diffusion-based text-to-image models generate unsafe content . experimental results show that all tested models suffer from unsafe generation . |
| Approach: | They propose a jailbreak that triggers diffusion-based text-to-image models to generate the image with visual text, resulting in unsafe content. |
| Outcome: | The proposed model generates image with visual text, but the model is unsafe under such jailbreak. |
Jailbreaking Safeguarded Text-to-Image Models via Large Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Text-to-image models generate harmful content when unsafe prompts are submitted . authors propose a method to jailbreak text-to image models with safety guardrails . |
| Approach: | They propose a method to jailbreak text-to-image models with safety guardrails . they use a fine-tuned large language model to generate adversarial prompts based on unsafe prompts. |
| Outcome: | The proposed method bypasses safety guardrails and outperforms existing no-box attacks . the proposed method generates adversarial prompts efficiently after fine-tuning the model . |
Chain-of-Jailbreak Attack for Image Generation Models via Step by Step Editing (2025.findings-acl)
Copied to clipboard
Wenxuan Wang, Kuiyi Gao, Youliang Yuan, Jen-tse Huang, Qiuzhi Liu, Shuai Wang, Wenxiang Jiao, Zhaopeng Tu
| Challenge: | Text-based image generation models, such as Stable Diffusion and DALL-E 3, hold significant potential in content creation and publishing workflows . however, considerable efforts are being made to prevent the generation of harmful content, such abusive, violent, or pornographic material. |
| Approach: | They propose a chain-of-jailbreak method which decomposes malicious queries into multiple sub-queries and iteratively edits images based on these sub-questions. |
| Outcome: | The proposed method can bypass safeguards of image generation models for over 60% cases, significantly outperforms other jailbreaking methods (14%) |
DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are susceptible to generating harmful content when prompted with carefully crafted inputs, a vulnerability known as LLM jailbreaking. |
| Approach: | They propose an end-to-end generative approach for jailbreak rewriting inspired by diffusion models that uses a sequence-tosequence (seq2sequ) diffusion model as a generator, conditioning on the original prompt and guiding the denoising process with a novel attack loss. |
| Outcome: | Experiments on Advbench and Harmbench show that the proposed method outperforms autoregressive jailbreak models across evaluation metrics including ASR, fluency, diversity and diversity. |
Adversarial Attacks on Parts of Speech: An Empirical Study in Text-to-Image Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that text-to-image models are vulnerable to adversarial perturbations . |
| Approach: | They investigate the impact of adversarial attacks on different POS tags within text prompts on T2I models. |
| Outcome: | The proposed model is vulnerable to adversarial perturbations with noun perturbations in text prompts. |
Universal Prompt Optimizer for Safe Text-to-Image Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies based on image checker, model fine-tuning and embedding blocking are impractical in real-world applications. |
| Approach: | They propose a novel reward function measuring toxicity and text alignment of generated images and train the optimizer through Proximal Policy Optimization. |
| Outcome: | The proposed model reduces the likelihood of various models in generating inappropriate images, with no significant impact on text alignment. |
Activation-Guided Local Editing for Jailbreaking Attacks (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for jailbreaking Large Language Models (LLMs) are limited and produce incoherent or unreadable inputs. |
| Approach: | They propose a two-stage framework that performs a one-shot, scenario-based generation of context and rephrases the original malicious query to obscure its harmful intent. |
| Outcome: | The proposed framework achieves state-of-the-art Attack Success Rate, with gains of up to 37.74% over the strongest baseline, and excellent transferability to black-box and large-scale models. |
Harmful Prompt Laundering: Jailbreaking LLMs with Abductive Styles and Symbolic Encoding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities but their misuse for harmful purposes remains a concern. |
| Approach: | They propose a jailbreaking technique that exploits weaknesses in LLMs' architecture . they propose abductive framing and symbolic encoding to bypass safeguards . |
| Outcome: | The proposed technique achieves over 95% attack success rate on GPT-series models and 70% across all targets. |
from Benign import Toxic: Jailbreaking the Language Model via Adversarial Metaphors (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have exposed the risk of Large Language Models (LLMs) generating harmful content by jailbreak attacks. |
| Approach: | They propose a framework that exploits AdVersArial meTAphoR to induce LLMs to calibrate harmful metaphors for jailbreaking. |
| Outcome: | The proposed framework can successfully jailbreak Large Language Models (LLMs) by leveraging the AdVersArial meTAphoR (AVATAR) framework achieves state-of-the-art attack success rate across multiple advanced LLMs. |
Red-Teaming NSFW Image Classifiers as Text-to-Image Safeguards (2026.findings-acl)
Copied to clipboard
Tinghao Xie, Yueqi Xie, Alireza Zareian, Shuming Hu, Felix Juefei-Xu, Xiaowen Lin, Ankit Jain, Prateek Mittal, Li Chen
| Challenge: | Not Safe for Work (NSFW) image classifiers play a critical role in safeguarding text-to-image systems. |
| Approach: | They propose an automated red-teaming framework that leverages a set of generative AI tools to uncover NSFW image failures. |
| Outcome: | The proposed framework uncovers and interprets failure modes and enables it to be applied to real-world T2I and T2V systems. |