| Challenge: | In-context representation hijacking attacks on large language models are insufficient to prevent harm, yet the mechanisms underlying this behavior remain unclear. |
| Approach: | They propose a simple in-context representation hijacking attack that replaces a harmful keyword with a benign token, and embeds the harmful semantics under a euphemism. |
| Outcome: | The proposed attack is optimized for open-source and achieves strong success rates on closed-source systems, reaching 74% on Llama-3.3-70B-Instruct with a single-sentence context override. |
Similar Papers
Reasoning Hijacking: The Fragility of Reasoning Alignment in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Current LLM safety research focuses on mitigating **Goal Hijacking**, preventing attackers from redirecting a model’s high-level objective. |
| Approach: | They propose a new adversarial prompt attack paradigm that subverts model judgments by injecting spurious decision criteria without altering the high-level task goal. |
| Outcome: | The proposed model subverts model judgments by injecting spurious decision criteria without altering the high-level task goal. |
Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attack (2024.acl-long)
Copied to clipboard
| Challenge: | Recent developments in balancing usefulness and safety of large language models raise a critical question . current attacks, especially adversarial ones that manipulate malicious prompts, often aim to manipulate the input . |
| Approach: | They show that LLMs can effectively summarize malicious long documents but often refuse to translate them. |
| Outcome: | The findings highlight a vulnerability in LLMs that can't translate or summarize documents . the study focuses on LLM models, Gemini and GPT-4, which can' be exploited . |
Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination (2026.acl-long)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) are capable of processing visual inputs, but are susceptible to hallucinations. |
| Approach: | They propose a method to localize and localize specific visual tokens, which are defined as **Inert Tokens**, across layers, revealing a rigid semantic collapse. |
| Outcome: | The proposed approach reduces the likelihood of LVLMs being hijacked by visual inputs while maintaining general capabilities. |
Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are susceptible to a type of attack known as jailbreaking, which misleads LLMs to output harmful contents. |
| Approach: | They propose to leverage hidden representations into existing jailbreak targets to move the attacks along the acceptance direction. |
| Outcome: | The proposed methods are validated using the objective of existing jailbreak attacks. |
XOXO: Stealthy Cross-Origin Context Poisoning Attacks against AI Coding Assistants (2026.acl-long)
Copied to clipboard
| Challenge: | Cross-Origin Context Poisoning (XOXO) exploits this automatic context inclusion by subtly manipulating code without changing its semantics. |
| Approach: | They propose a novel attack that exploits automatic context inclusion by subtly manipulating code without changing its semantics. |
| Outcome: | The proposed attack achieves 73.20% success rates against eight state-of-the-art models including GPT 4.1 and Claude 3.5 Sonnet v2 and vulnerability injection rates up to 66.67%. |
Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning has shown high efficacy in several NLP tasks, especially in few-shot settings. |
| Approach: | They propose a backdoor attack method that poisons demonstration examples and poisons the demonstration context, preserving the model's generality. |
| Outcome: | The proposed method can make models behave in alignment with predefined intentions without fine-tuning the model. |
Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Disconnect between tokenizer creation and model training in language models allows for specific inputs, such as the infamous SolidGoldMagikarp token, to induce unwanted model behaviour. |
| Approach: | They propose to use tokenizer analysis, model weight-based indicators, and prompting techniques to detect problematic tokens in large language models. |
| Outcome: | The proposed methods show that tokenizers are under-trained across a diverse set of models and provide insights into improving the efficiency and safety of language models. |
Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Text-to-image (T2I) models can be used to generate harmful content such as sexually explicit, unfaithful, and misleading or Not-Safe-for-Work (NSFW) images. |
| Approach: | They propose a more practical and universal attack that does not require the presence of a target model. |
| Outcome: | The proposed attack bypasses both text and image safety checkers while preserving high semantic alignment with the target prompt. |
SequentialBreak: Large Language Models Can be Fooled by Embedding Jailbreak Prompts into Sequential Prompt Chains (2025.acl-srw)
Copied to clipboard
| Challenge: | SequentialBreak enables LLMs to bypass safety mechanisms by arranging malicious prompts in a single query. |
| Approach: | They propose a single-query jailbreak technique that arranges multiple benign prompts in sequence with a hidden malicious instruction among them to bypass safety mechanisms. |
| Outcome: | The proposed technique outperforms baselines on open-source and closed-source models. |
Into the Gray Zone: Domain Contexts Can Blur LLM Safety Boundaries (2026.acl-long)
Copied to clipboard
Ki Sen Hung, Xi Yang, Chang Liu, Haoran Li, Kejiang Chen, Changxuan Fan, Tsun On Kwok, Weiming Zhang, Xiaomeng Li, Yangqiu Song
| Challenge: | a goal of LLM alignment is to balance usefulness with harmlessness, but this conflictes when knowledge serves both legitimate and malicious purposes. |
| Approach: | They propose a framework that combines safety-research contexts with adversarial interactions to exploit a vulnerability in Jargon queries. |
| Outcome: | a framework outperforms existing methods in analyzing Jargon queries, a study shows . it achieves 93% of attacks across seven models, while remaining useful, the authors say . |