A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for generating 'jailbreaks' suffer from manual design or require optimization on other white-box models, which compromises either generalization or efficiency. |
| Approach: | They propose a framework that leverages LLMs to generate effective jailbreak prompts and a generalized framework that can be used to generate prompts. |
| Outcome: | The proposed framework improves the attack success rate while reducing the time cost compared to baselines. |
Similar Papers
Defensive Prompt Patch: A Robust and Generalizable Defense of Large Language Models against Jailbreak Attacks (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have showcased their ability to understand and generate text akin to human interaction. |
| Approach: | They propose a prompt-based defense mechanism specifically designed to protect LLMs against jailbreak attacks by introducing jailbreak prompts into malicious queries. |
| Outcome: | Empirical results show that the proposed defense outperforms existing defense strategies in balancing safety and utility while maintaining high utility. |
A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated capabilities for generating content that could be deemed harmful. |
| Approach: | They conduct a comprehensive analysis of existing studies on jailbreaking LLMs and their defense techniques. |
| Outcome: | The proposed techniques underperform existing white-box attacks and include special tokens significantly affects the likelihood of successful attacks. |
Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to jailbreak large language models have been poorly studied . a recent study showed that non-expert users can jailbreak LLMs by manipulating their prompts . |
| Approach: | They propose a formalism and a taxonomy of known (and possible) jailbreaks . they propose generating a dataset of model outputs across 3700 jailbreak prompts a 'prompt' attack is a new attack popularly categorized as "prompting injection attacks" |
| Outcome: | The proposed model exploits 3700 jailbreak prompts over 4 tasks to analyze their effectiveness . authors show that the model can learn to perform a new task on unseen examples . |
SQL Injection Jailbreak: A Structural Disaster of Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to jailbreak Large Language Models (LLMs) exploited internal properties or capabilities of the model, such as optimization-based jailbreak methods and methods that leveraged the model’s context-learning abilities. |
| Approach: | They propose a new method which injects jailbreak information into user prompts and induces the model to generate harmful content. |
| Outcome: | The proposed method achieves near 100% success rates on open-source models while incurring lower time costs compared to previous methods. |
Jailbreaking with Universal Multi-Prompts (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have seen rapid development in recent years, but ethical concerns and new types of attacks have emerged. |
| Approach: | They propose a prompt-based method to jailbreak large language models using universal multi-prompts and an approach for defense that outperforms existing techniques. |
| Outcome: | The proposed method outperforms existing techniques for jailbreaking LLMs using universal multi-prompts. |
Diversity Helps Jailbreak Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for jailbreaking large language models rely on laborious human engineering and whitebox access to model internals. |
| Approach: | They propose a method that instructs large language models to deviate from prior context and generate harmful outputs by instructing them to deviat from previous attacks. |
| Outcome: | The proposed method achieves a 62.83% higher success rate in compromising ten leading chatbots, while using only 12.9% of the queries. |
Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing defense methods focus on detecting harmful prompts or reducing the likelihood of harmful responses. |
| Approach: | They propose a layer-specific editing method to align LLMs to harmful prompts by supervised fine-tuning and reinforcement learning. |
| Outcome: | The proposed method improves the performance of large language models against jailbreak attacks while maintaining performance on benign prompts. |
Defending LLMs against Jailbreaking Attacks via Backtranslation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advancement in large language models (LLMs) has shown their extensive applications and transformative potential to reshape people's lives. |
| Approach: | They propose a method which uses backtranslation to infer an input prompt from an input input prompt and then run it again on the backtranslated prompt. |
| Outcome: | The proposed method outperforms baselines and has little impact on the generation quality for benign input prompts. |
Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are susceptible to a type of attack known as jailbreaking, which misleads LLMs to output harmful contents. |
| Approach: | They propose to leverage hidden representations into existing jailbreak targets to move the attacks along the acceptance direction. |
| Outcome: | The proposed methods are validated using the objective of existing jailbreak attacks. |
LLMs Caught in the Crossfire: Malware Requests and Jailbreak Challenges (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have a high vulnerability to jailbreak attacks that leverage crafted prompts to generate malicious outputs. |
| Approach: | They propose to use large language models to test their security against jailbreak attacks that leverage crafted prompts to generate malicious outputs. |
| Outcome: | The proposed model is based on 320 manually crafted malicious code generation requirements, covering 11 jailbreak methods and 29 code functionality categories. |