Jingyu Zhang, Ahmed Elgohary, Xiawei Wang, A S M Iftekhar, Ahmed Magooda, Benjamin Van Durme, Daniel Khashabi, Kyle Jackson
| Challenge: | a new framework for safety benchmarking is being developed for large language models . the framework "distills" jailbreak attacks into high-quality and easily-updatable safety benchmarks . |
| Approach: | They propose a framework that "distills" jailbreak attacks into high-quality safety benchmarks. |
| Outcome: | The proposed framework "distills" jailbreak attacks into high-quality safety benchmarks . it requires minimal human effort to rerun the pipeline and produce updated benchmarks. |
Similar Papers
Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models (2026.acl-long)
Copied to clipboard
Zirui Song, Qian Jiang, Mingxuan Cui, Mingzhe Li, Lang Gao, Zeyu Zhang, Zixiang Xu, Yanbo Wang, Guangxian Ouyang, Zhenhao Chen, Xiuying Chen
| Challenge: | a recent study evaluated large audio-language models against jailbreak attacks . a new benchmark is being developed to evaluate LAM safety against jailbreaking attacks based on temporal and semantic nature of speech . |
| Approach: | They propose a benchmark to evaluate LAM jailbreak vulnerabilities in adversarial audio prompts . they use a dataset of 1,495 adversarials to evaluate their performance . |
| Outcome: | The proposed benchmark evaluates state-of-the-art LAMs against jailbreak attacks . it demonstrates that even small, semantically preserved perturbations can reduce safety . |
EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models (2025.emnlp-demos)
Copied to clipboard
| Challenge: | In this paper, we present EasyDistill, a comprehensive toolkit designed for effective black-box and white-box knowledge distillation (KD) of large language models. |
| Approach: | They propose a toolkit for effective black-box and white-box knowledge distillation (KD) of large language models (LLMs). |
| Outcome: | The framework offers data synthesis, supervised fine-tuning, ranking optimization, and reinforcement learning techniques specifically tailored for KD scenarios. |
Beneath the Facade: Probing Safety Vulnerabilities in LLMs via Auto-Generated Jailbreak Prompts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a gap exists in systematic assessment of real-world safety risks . a lack of evaluation frameworks to keep pace with the breadth and variability of real risk factors. |
| Approach: | They propose a framework that assesses security vulnerabilities by deriving scenario-driven jailbreak prompts using an adversarial agent. |
| Outcome: | The proposed framework detects vulnerabilities in scenario-driven jailbreak prompt generation . it supports plug-and-play jailbreak strategies and adapts seamlessly to multimodal settings. |
LLMs Caught in the Crossfire: Malware Requests and Jailbreak Challenges (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have a high vulnerability to jailbreak attacks that leverage crafted prompts to generate malicious outputs. |
| Approach: | They propose to use large language models to test their security against jailbreak attacks that leverage crafted prompts to generate malicious outputs. |
| Outcome: | The proposed model is based on 320 manually crafted malicious code generation requirements, covering 11 jailbreak methods and 29 code functionality categories. |
Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring (2025.naacl-long)
Copied to clipboard
Honglin Mu, Han He, Yuxin Zhou, Yunlong Feng, Yang Xu, Libo Qin, Xiaoming Shi, Zeming Liu, Xudong Han, Qi Shi, Qingfu Zhu, Wanxiang Che
| Challenge: | Existing black-box jailbreak methods often rely on model feedback . existing methods may be intercepted by content moderators during the search process . |
| Approach: | They propose a method that guides malicious prompt construction by local training a mirror model of the target black-box model through benign data distillation. |
| Outcome: | The proposed method achieves a 92% attack success rate and 80% stealth rate on a subset of AdvBench. |
Stand on The Shoulders of Giants: Building JailExpert from Previous Attack Experience (2025.emnlp-main)
Copied to clipboard
Xi Wang, Songlei Jian, Shasha Li, Xiaopeng Li, Bin Ji, Ma Jun, Xiaodong Liu, Jing Wang, Jianfeng Zhang, Jie Yu, Feilong Bao, null Wangbaosheng
| Challenge: | Existing methods to generate human-aligned content with a “jailbreak prompt” are inefficient and repetitive, causing inefficiency and a lack of experience. |
| Approach: | They propose a framework that integrates past attack experiences to aid current jailbreak attempts. |
| Outcome: | The proposed framework improves both attack effectiveness and efficiency compared to the current black-box jailbreak method. |
Defensive Prompt Patch: A Robust and Generalizable Defense of Large Language Models against Jailbreak Attacks (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have showcased their ability to understand and generate text akin to human interaction. |
| Approach: | They propose a prompt-based defense mechanism specifically designed to protect LLMs against jailbreak attacks by introducing jailbreak prompts into malicious queries. |
| Outcome: | Empirical results show that the proposed defense outperforms existing defense strategies in balancing safety and utility while maintaining high utility. |
SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding (2024.acl-long)
Copied to clipboard
| Challenge: | Despite advances in large language models, they face substantial challenges in terms of safety. |
| Approach: | They develop a safety-aware decoding strategy for large language models to defend against jailbreak attacks. |
| Outcome: | The proposed strategy outperforms six defense methods against jailbreak attacks on five LLMs. |
From Attack Surfaces to Actual Operations: A Survey of Modern LLM Jailbreaks (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing taxonomies focus on manipulation methods rather than underlying mechanisms, limiting our understanding of attack effectiveness and defensive strategies. |
| Approach: | They propose a two-fold taxonomy to categorize attacks across three tiers based on exploited vulnerabilities and approaches and an operational taxonomies to evaluate attacks across four dimensions. |
| Outcome: | The proposed taxonomy categorizes attacks across three tiers based on exploited vulnerabilities and approaches and evaluates attacks on four dimensions to assess real-world feasibility and sustainability. |
Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to jailbreak large language models have been poorly studied . a recent study showed that non-expert users can jailbreak LLMs by manipulating their prompts . |
| Approach: | They propose a formalism and a taxonomy of known (and possible) jailbreaks . they propose generating a dataset of model outputs across 3700 jailbreak prompts a 'prompt' attack is a new attack popularly categorized as "prompting injection attacks" |
| Outcome: | The proposed model exploits 3700 jailbreak prompts over 4 tasks to analyze their effectiveness . authors show that the model can learn to perform a new task on unseen examples . |