GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for detecting jailbreak prompts are primarily online moderation APIs or finetuned LLMs. |
| Approach: | They propose a method which scrutinizes the gradients of safety-critical parameters in large LLMs to detect jailbreak prompts. |
| Outcome: | The proposed method outperforms Llama Guard in detecting jailbreak prompts despite extensive finetuning with a large dataset. |
Similar Papers
SafeQuant: LLM Safety Analysis via Quantized Gradient Inspection (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to jailbreak Large Language Models (LLMs) use computationally intensive verification or require adversarial fine-tuning, leaving models vulnerable to advanced attacks. |
| Approach: | They propose a framework that leverages quantized gradient patterns to identify harmful prompts efficiently. |
| Outcome: | The proposed framework outperforms existing defenses across multiple benchmarks while maintaining model utility. |
A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for generating 'jailbreaks' suffer from manual design or require optimization on other white-box models, which compromises either generalization or efficiency. |
| Approach: | They propose a framework that leverages LLMs to generate effective jailbreak prompts and a generalized framework that can be used to generate prompts. |
| Outcome: | The proposed framework improves the attack success rate while reducing the time cost compared to baselines. |
LLM Jailbreak Detection for (Almost) Free! (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for detecting jailbreak prompts entail significant computational costs . |
| Approach: | They propose a free jailbreak detection method which scales logits by temperature to detect jailbreak prompts . |
| Outcome: | The proposed method detects jailbreak prompts with no additional computational costs. |
Why Not Act on What You Know? Unleashing Safety Potential of LLMs via Self-Aware Guard Enhancement (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive capabilities across various tasks but are vulnerable to meticulously crafted jailbreak attacks. |
| Approach: | They propose a training-free defense strategy to align LLMs’ strong safety discrimination performance with their relatively weaker safety generation ability. |
| Outcome: | The proposed strategy achieves an average 99% success rate against numerous complex and covert jailbreak methods while maintaining helpfulness on general benchmarks. |
RLShield: Dynamic Jailbreak Detection for LLMs via Reinforced Adaptive Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to detect jailbreak prompts rely on static model components or fixed decision thresholds. |
| Approach: | They propose a dynamic jailbreak detection framework that employs reinforcement learning for adaptive threshold selection. |
| Outcome: | Experimental results show that the framework outperforms baselines in detection performance while maintaining high computational efficiency. |
Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to jailbreak large language models have been poorly studied . a recent study showed that non-expert users can jailbreak LLMs by manipulating their prompts . |
| Approach: | They propose a formalism and a taxonomy of known (and possible) jailbreaks . they propose generating a dataset of model outputs across 3700 jailbreak prompts a 'prompt' attack is a new attack popularly categorized as "prompting injection attacks" |
| Outcome: | The proposed model exploits 3700 jailbreak prompts over 4 tasks to analyze their effectiveness . authors show that the model can learn to perform a new task on unseen examples . |
Jailbreaking Attacks vs. Content Safety Filters: How Far Are We in the LLM Safety Arms Race? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have focused on the models, neglecting the full deployment pipeline . previous studies have underestimated the practical success of these attacks . |
| Approach: | They evaluate the effectiveness of jailbreak attacks targeting LLM safety alignment . they highlight critical gaps and call for further refinement of detection accuracy and usability . |
| Outcome: | The proposed attacks can detect at least one safety filter across the entire deployment pipeline. |
Jailbreak Attack Initializations as Extractors of Compliance Directions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Safety-aligned LLMs respond to prompts with compliance or refusal, each corresponding to distinct directions in the model’s activation space. |
| Approach: | They propose an initialization framework that aims to project unseen prompts further along compliance directions. |
| Outcome: | The proposed initialization framework achieves an increased attack success rate and reduced computational overhead, highlighting the fragility of safety-aligned LLMs. |
When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models (2026.eacl-srw)
Copied to clipboard
| Challenge: | Existing safety evaluations rely on fixed collections of harmful prompts . such attacks span single-shot prompts, multi-turn interactions, cross-lingual settings . |
| Approach: | They propose to use black-box prompt optimization techniques to search for safety failures . they use GPT-5.1 to optimize for a continuous danger score . |
| Outcome: | The proposed approach reduces effective safeguards for large language models . the average danger score of Qwen 3 8B increases from 0.09 in its baseline setting to 0.79 after optimization. |
SHARP: Self-adaptive Harmful Category-aware Prompt Generation for Black-box Jailbreaking (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for jailbreak ignore the semantic differences between categories of harmful questions, leading to inconsistent success rates and reduced overall attack effectiveness. |
| Approach: | They propose a category-aware jailbreak framework that incorporates the semantic category of harmful questions into prompt generation. |
| Outcome: | The proposed framework improves attack success rates and category alignment and achieves better cross-category robustness compared to the state-of-the-art (SOTA) baselines. |