Papers with AdvBench
The Unintended Trade-off of AI Alignment: Balancing Hallucination Mitigation and Safety in LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Hallucination in large language models has been studied, but a side effect remains unrecognized . a new study examines the trade-off between truthfulness and safety alignment . |
| Approach: | They propose a method that disentangles hallucination from hallucinian features using sparse autoencoders. |
| Outcome: | The proposed method preserves refusal behavior and task utility while maintaining safety alignment. |
Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring (2025.naacl-long)
Copied to clipboard
Honglin Mu, Han He, Yuxin Zhou, Yunlong Feng, Yang Xu, Libo Qin, Xiaoming Shi, Zeming Liu, Xudong Han, Qi Shi, Qingfu Zhu, Wanxiang Che
| Challenge: | Existing black-box jailbreak methods often rely on model feedback . existing methods may be intercepted by content moderators during the search process . |
| Approach: | They propose a method that guides malicious prompt construction by local training a mirror model of the target black-box model through benign data distillation. |
| Outcome: | The proposed method achieves a 92% attack success rate and 80% stealth rate on a subset of AdvBench. |
\mathsf{Con Instruction}: Universal Jailbreaking of Multimodal Large Language Models via Non-Textual Modalities (2025.acl-long)
Copied to clipboard
| Challenge: | Existing attacks communicate instruction through text, accompanied by a toxic image or audio . a novel gray-box attack method generates adversarial images or audio to convey harmful instructions to MLLMs . |
| Approach: | They propose a gray-box attack method that generates adversarial images or audio to convey specific harmful instructions to MLLMs by following non-textual instruction. |
| Outcome: | The proposed method achieves highest success rates on visual and audio-language models . larger models are more susceptible toCon Instruction, compared to their underlying models - the results will be released . |
Atoxia: Red-teaming Large Language Models with Target Toxic Answers (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are still vulnerable to generation safety vulnerabilities. |
| Approach: | They propose a method that A**tacks LLMs with target "toxi" given a particular harmful answer, the method generates a user query and a misleading answer opening to examine the internal defects of a given LLM. |
| Outcome: | The proposed method detects safety risks in open-source models and state-of-the-art models such as GPT-4o. |
Multilingual Collaborative Defense for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing safeguards for Large Language Models are vulnerable to "jailbreaking" harmful queries. |
| Approach: | They propose a learning method that optimizes a continuous soft safety prompt automatically to facilitate multilingual safeguarding of LLMs. |
| Outcome: | The proposed method outperforms previous approaches in multilingual jailbreak defense while exhibiting strong cross-lingual generalization. |
Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated increasing power, but they also have vulnerabilities. |
| Approach: | They propose a black-box attack that targets the cognitive structure and processes of large language models (LLMs) they propose defending cognitive overload attacks from three perspectives. |
| Outcome: | The proposed attack is a black-box attack with no need for knowledge of model architecture or access to model weights. |
SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding (2024.acl-long)
Copied to clipboard
| Challenge: | Despite advances in large language models, they face substantial challenges in terms of safety. |
| Approach: | They develop a safety-aware decoding strategy for large language models to defend against jailbreak attacks. |
| Outcome: | The proposed strategy outperforms six defense methods against jailbreak attacks on five LLMs. |
Exploiting the Index Gradients for Optimization-Based Jailbreaking on Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Despite advances in training Large Language Models, they remain vulnerable to jailbreak, an adversarial attack method. |
| Approach: | They propose an adversarial jailbreak algorithm that exploits the gradient information of the suffix tokens to accelerate the optimization process. |
| Outcome: | The proposed model achieves 1.5x speedup while maintaining high attack success rates. |
Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing alignment methods focus on reactive feedback, where immediate human perception is leveraged to judge sampled model responses as preference data for post-training. |
| Approach: | They propose a proof-of-concept framework that projects how model-generated advice could propagate through societal systems on a macroscopic scale over time, enabling more robust alignment. |
| Outcome: | The proposed framework achieves 20% improvement on existing safety benchmarks and an average win rate exceeding 70% against strong baselines. |
SQL Injection Jailbreak: A Structural Disaster of Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to jailbreak Large Language Models (LLMs) exploited internal properties or capabilities of the model, such as optimization-based jailbreak methods and methods that leveraged the model’s context-learning abilities. |
| Approach: | They propose a new method which injects jailbreak information into user prompts and induces the model to generate harmful content. |
| Outcome: | The proposed method achieves near 100% success rates on open-source models while incurring lower time costs compared to previous methods. |
A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaos (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Reasoning Models (LRMs) have advanced beyond traditional Large Language Models, yet they pose heightened safety risks. |
| Approach: | They propose a first jailbreak attack targeting Large Reasoning Models . they exploit a Chaos Machine component to transform attack prompts with diverse one-to-one mappings based on the reasoning chain . |
| Outcome: | The proposed attack exploits the unique vulnerabilities of LRMs by integrating a Chaos Machine. success rates of the mousetrap attack are as high as 96%, 86% and 98% respectively. |
Gamma-Guard: Lightweight Residual Adapters for Robust Guardrails in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are widely deployed as zero-shot evaluators for answer grading, content moderation, and document ranking. |
| Approach: | They propose a system that trains LLMs with adapters to denoise embeddings and refocus attention. |
| Outcome: | The proposed model lifts adversarial accuracy from 5% to 95% a 90 percentage-point gain while reducing clean-data accuracy by just 8 percentage points. |
Attention Eclipse: Manipulating Attention to Bypass LLM Safety-Alignment (2025.emnlp-main)
Copied to clipboard
Pedram Zaree, Md Abdullah Al Mamun, Quazi Mishkatul Alam, Yue Dong, Ihsen Alouani, Nael Abu-Ghazaleh
| Challenge: | Recent research has shown that carefully crafted jailbreak inputs can induce large language models to produce harmful outputs, despite safety measures such as alignment. |
| Approach: | They propose a method for generating highly effective Jailbreak attacks that selectively strengthen or weaken attention among different parts of the prompt. |
| Outcome: | The proposed attacks amplify the success rate of existing Jailbreak algorithms while lowering generation cost. |
PROOD: A Simple LLM Out-of-Distribution Guardrail Leveraging Response Semantics (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing OOD methods often struggle with deliberately obfuscated, context-dependent, or superficially benign prompts. |
| Approach: | They propose a framework that jointly analyzes LLM prompts and their outputs to improve semantic understanding. |
| Outcome: | The proposed framework outperforms existing OOD methods on three benchmarks and improves F1 scores by up to 6.3 points. |
AutoRAN: Automated Hijacking of Safety Reasoning in Large Reasoning Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large reasoning models (LRMs) generate coherent reasoning paths before conclusions, but they introduce new vulnerabilities. |
| Approach: | They propose a framework that leverages a weaker but less-aligned model to simulate execution reasoning for initial hijacking attempts and iteratively refines attacks by exploiting reasoning patterns leaked through the target LRM’s refusals. |
| Outcome: | The proposed framework achieves 100% success rate within one or few turns, neutralizing reasoning-based defenses even when evaluated by robustly aligned external models. |
Red-Bandit: Test-Time Adaptation for LLM Red-Teaming via Bandit-Guided LoRA Experts (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to audit Large Language Models (LLMs) lack mechanisms to efficiently adapt to model-specific vulnerabilities at inference. |
| Approach: | They propose a red-teaming framework that adapts online to identify and exploit model failure modes under distinct attack styles. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on AdvBench and HarmBench, while generating more human-readable adversarial prompts (lower perplexity). |