False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work has leveraged probing-based approaches to study the separability of malicious and benign inputs in Large Language Models’ internal representations. |
| Approach: | They propose to use probing-based methods to study separability of malicious and benign inputs in LLMs' internal representations to detect harmful and benign content. |
| Outcome: | The proposed methods show that they learn superficial patterns rather than semantic harmfulness. |
Similar Papers
Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs (2025.findings-emnlp)
Copied to clipboard
Yu Yan, Sheng Sun, Zhe Wang, Yijun Lin, Zenghao Duan, Zhifei Zheng, Min Liu, Zhiyi Yin, Jianping Zhang
| Challenge: | Large Language Models have been developed to deal with real-world crimes, but it remains unclear whether they internalize authentic knowledge or are forced to simulate toxic language patterns. |
| Approach: | They construct knowledge-intensive Q&A to investigate misuse threats of Large Language Models in terms of dangerous knowledge possession, harmful task planning utility, and harmfulness judgment robustness. |
| Outcome: | The findings raise concerns that jailbreak success is often attributable to a hallucination loop between jailbroken LLM and judger LLM . |
Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional proficiency in instruction-following, making them increasingly integral to various applications. |
| Approach: | They establish a benchmark to evaluate the robustness of instruction-following LLMs against prompt injection attacks, assessing their ability to discern which instructions to follow and which to disregard. |
| Outcome: | The proposed model is overly sensitive to prompt injection attacks, focusing on the latter part of the prompt without fully understanding the context. |
Stumbling Blocks: Stress Testing the Robustness of Machine-Generated Text Detectors Under Attacks (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies on this topic focus on the robustness of specific detectors or particular attack methods. |
| Approach: | They stress test the detectors’ robustness to malicious attacks under realistic scenarios using LLMs and metric-based detectors. |
| Outcome: | The proposed methods are based on a set of LLM-based models and their performance is compared under different budget levels. |
The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts (2024.findings-acl)
Copied to clipboard
Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, Daniel Khashabi
| Challenge: | Recent studies show that malicious prompt instructions could solicit objectionable content from LLMs. |
| Approach: | They compare how state-of-the-art LLMs respond to malicious prompts in different languages . they find that LLM's generate unsafe responses more often when a prompt is written in a lower-resource language . |
| Outcome: | The proposed model can generate unsafe responses more often when a malicious prompt is written in a lower-resource language, and less irrelevant responses when written in lower-source languages. |
What Does Parameter-free Probing Really Uncover? (2024.acl-short)
Copied to clipboard
| Challenge: | Probing large language models (LLMs) has been criticized for using pre-defined label-laden target labels. |
| Approach: | They extend a parameter-free probing technique called perturbed masking applied to BERT to examine the relationship between UD and BERT. |
| Outcome: | The proposed method is compared to the UD formalism for English and shows that it lacks correlations with linguistic theory. |
How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilities (2024.naacl-long)
Copied to clipboard
| Challenge: | Rapid progress in open-source Large Language Models (LLMs) is driving AI development, but lacks sufficient trustworthiness to detect and mitigate adversarial demonstrations. |
| Approach: | They propose an extended Chain of Utterances-based (CoU) prompting strategy to attack open-source LLMs. |
| Outcome: | The proposed attack strategy is based on malicious demonstrations and toxicity tests on open-source models. |
Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction (2026.findings-acl)
Copied to clipboard
Yulin Chen, Haoran Li, Yuan Sui, Yue Liu, Yufei He, Xiaoling Bai, Chi Fei, Li Yabo, Haozhe Ma, Yangqiu Song, Bryan Hooi
| Challenge: | Prompt injection attacks manipulate large language models (LLMs) by misleading them to deviate from the original input instructions and execute maliciously injected instructions. |
| Approach: | They propose a prompt injection defense method that suppresses the model's instruction-following tendencies rather than suppressing them. |
| Outcome: | The proposed method outperforms prompt-engineering-based approaches and fine-tuning methods and reduces the ASR to nearly 0% in some scenarios. |
LLM Sensitivity Challenges in Abusive Language Detection: Instruction-Tuned vs. Human Feedback (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies show that instruction-tuned LLMs under-predict positive classes . however, they are overly sensitive and can be applied for abuse detection without fine-tuning . |
| Approach: | They show that instruction-tuned LLMs tend to under-predict positive classes . they also show that label frequency in the prompt helps with the significant over-prediction . |
| Outcome: | The proposed models under-predict positive classes in social media, whereas they are overly sensitive. |
Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are susceptible to a type of attack known as jailbreaking, which misleads LLMs to output harmful contents. |
| Approach: | They propose to leverage hidden representations into existing jailbreak targets to move the attacks along the acceptance direction. |
| Outcome: | The proposed methods are validated using the objective of existing jailbreak attacks. |
Do We Know What LLMs Don’t Know? A Study of Consistency in Knowledge Probing (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for probing knowledge gaps in large language models are inconsistent and inconsistent. |
| Approach: | They propose a process based on input variations and quantitative metrics to evaluate probing methods that are inconsistent on knowledge gaps. |
| Outcome: | The proposed process exposes two dimensions of inconsistency in knowledge gap probing. |