How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) rely on safety alignment to avoid malicious user inputs. |
| Approach: | They employ weak classifiers to explain LLM safety through the intermediate hidden states. |
| Outcome: | The proposed model can identify malicious and normal inputs and detect malicious ones without jailbreak. |
Similar Papers
Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Understanding how jailbreaking works remains limited, hindering the development of effective defense strategies. |
| Approach: | They propose a new mechanism that adaptively constrains activations within the safety boundary and propose 'Activation Boundary Defense' to enhance its effectiveness. |
| Outcome: | The proposed defense achieves an average Defense Success Rate (DSR) of over 98% against various jailbreak attacks, with less than 2% impact on the model’s general capabilities. |
Jailbreaks as Inference-Time Alignment: A Framework for Understanding Safety Failures in LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models are safety-aligned to prevent harmful response generation . prior work on jailbreak effectiveness has focused on analyzing success rate of jailbreaks . |
| Approach: | They propose to frame jailbreaks as inference-time alignment and draw suboptimal bounds . they also propose a Safety-Net to measure how vulnerable an LLM is to jailbreak attacks . |
| Outcome: | a new framework allows researchers to show how vulnerable an LLM is to jailbreaks . a Safety-Net measures how vulnerable the model is to attacks, the authors say . |
Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are susceptible to a type of attack known as jailbreaking, which misleads LLMs to output harmful contents. |
| Approach: | They propose to leverage hidden representations into existing jailbreak targets to move the attacks along the acceptance direction. |
| Outcome: | The proposed methods are validated using the objective of existing jailbreak attacks. |
Why Safeguarded Ships Run Aground? Aligned Large Language Models’ Safety Mechanisms Tend to Be Anchored in The Template Region (2025.acl-long)
Copied to clipboard
| Challenge: | Infilling a fixed template between the input instruction and initial model output is a common practice for existing LLMs, but it is vulnerable to inference-time jailbreak attacks. |
| Approach: | They propose to fill a fixed template between the input instruction and initial model output and to detach safety mechanisms from the template region to mitigate the risk of inference-time jailbreak attacks. |
| Outcome: | The proposed method is widespread across aligned LLMs and shows that it mitigates inference-time jailbreak vulnerabilities. |
On the Vulnerability of Safety Alignment in Open-Access LLMs (2024.findings-acl)
Copied to clipboard
Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, Fangzhao Wu
| Challenge: | Large language models (LLMs) are susceptible to malicious exploitation, but are often rejected and limited harmfulness is limited. |
| Approach: | They propose two types of reverse alignment techniques: reverse supervised fine-tuning (RSFT) and reverse preference optimization (RPO). |
| Outcome: | The proposed methods can significantly enhance the success rate and harmfulness of jailbreak attacks, but they face high rejection rates and limited harmfulness. |
SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) with safe-alignment training are vulnerable to jailbreak attacks, causing malicious users to generate harmful outputs. |
| Approach: | They propose a safe-alignment jailbreak method that bypasses the middle-to-late layers of large language models by a residual connection. |
| Outcome: | The proposed method improves by 51% over the best performing baseline GCG on HarmBench test set. |
Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs (2025.findings-emnlp)
Copied to clipboard
Yu Yan, Sheng Sun, Zhe Wang, Yijun Lin, Zenghao Duan, Zhifei Zheng, Min Liu, Zhiyi Yin, Jianping Zhang
| Challenge: | Large Language Models have been developed to deal with real-world crimes, but it remains unclear whether they internalize authentic knowledge or are forced to simulate toxic language patterns. |
| Approach: | They construct knowledge-intensive Q&A to investigate misuse threats of Large Language Models in terms of dangerous knowledge possession, harmful task planning utility, and harmfulness judgment robustness. |
| Outcome: | The findings raise concerns that jailbreak success is often attributable to a hallucination loop between jailbroken LLM and judger LLM . |
Unraveling LLM Jailbreaks Through Safety Knowledge Neurons (2026.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have achieved significant progress in alignment, ensuring safer and more reliable outputs. |
| Approach: | They propose a neuron-level interpretability method that focuses on the role of safety-related knowledge neurons to improve model robustness against jailbreak attacks. |
| Outcome: | The proposed method reduces attack success rates across multiple LLMs and outperforms all baseline defenses. |
Unleashing the Unseen: Harnessing Benign Datasets for Jailbreaking Large Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Despite significant efforts in safety alignment, large language models (LLMs) such as GPT-4 and LLaMA 3 remain vulnerable to jailbreak attacks that can induce harmful behaviors. |
| Approach: | They propose a feature extraction method to extract sample-agnostic features from benign datasets in the form of adversarial suffixes and propose 'suffix maybe features' they show that adversarials generated from jailbreak attacks may contain meaningful features, i.e. appending the same suffix to different prompts results in responses exhibiting specific characteristics. |
| Outcome: | The proposed method extracts sample-agnostic features from benign datasets and shows that they may contain meaningful features. |
Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant advancements but can be misused to generate harmful content. |
| Approach: | They propose a Robustly Aligned LLM to defend against alignment-breaking attacks by retraining existing LLMs and using adversarial or handcrafted jailbreaking prompts. |
| Outcome: | The proposed model reduces attack success rates from nearly 100% to around 10% or less. |