TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Model (LLM) services and models often come with legal rules on who can use them and how they must use them. |
| Approach: | They propose a method that uses adversarial suffixes to get an answer from a target LLM. |
| Outcome: | The proposed method detects the LLMs with over 95% true positive rate at under 0.2% false positive rate even after a single interaction. |
Similar Papers
Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs Through a Global Prompt Hacking Competition (2023.emnlp-main)
Copied to clipboard
Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis-François Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Kost, Christopher Carnahan, Jordan Boyd-Graber
| Challenge: | Large Language Models are increasingly being deployed in interactive contexts that involve direct user engagement. |
| Approach: | They run a global prompt hacking competition to encourage research on prompt hacks . they elicit 600K+ adversarial prompts against three state-of-the-art LLMs based on a dataset . |
| Outcome: | The results of the competition show that current LLMs can be manipulated via prompt hacking . the competition elicits 600K+ adversarial prompts against three state-of-the-art LLM models . |
The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | cipher decoding, riddles, code execution embedded into model prompts bypass safety safeguards of large language models (LLMs) . |
| Approach: | They introduce a novel class of adversarial jailbreak adversarials on large language models, termed Task-in-Prompt (TIP) attacks. |
| Outcome: | The proposed techniques circumvent safeguards in six state-of-the-art language models, including GPT-4o and LLaMA 3.2, and consistently generate restricted content . |
PROMPRINT: Prompt Fingerprinting via First-Token Response for LLM App Cloning Detection (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Model apps are increasingly regarded as intellectual property . cloned apps pose risks of copyright infringement and malicious misuse . |
| Approach: | They propose a fingerprinting approach that optimizes queries that induce the LLM to generate a specific first token associated with the given system prompt. |
| Outcome: | The proposed fingerprinting approach is robust to partial system prompt modifications and effective under injection of adversarial instructions. |
Diversity Helps Jailbreak Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for jailbreaking large language models rely on laborious human engineering and whitebox access to model internals. |
| Approach: | They propose a method that instructs large language models to deviate from prior context and generate harmful outputs by instructing them to deviat from previous attacks. |
| Outcome: | The proposed method achieves a 62.83% higher success rate in compromising ten leading chatbots, while using only 12.9% of the queries. |
Monte Carlo Tree Search Based Prompt Autogeneration for Jailbreak Attacks against LLMs (2025.coling-main)
Copied to clipboard
| Challenge: | Jailbreak attacks craft specific prompts or append adversarial suffixes to prompts, thereby inducing language models to generate harmful or unethical content and bypassing the model’s safety guardrails. |
| Approach: | They propose a Monte Carlo Tree Search (MCTS) based Prompt Auto-generation (MPA) method to generate adversarial suffixes for valid jailbreak attacks. |
| Outcome: | The proposed method outperforms existing methods on open-source and closed-source models and shows that it can generate harmful responses. |
Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to jailbreak large language models have been poorly studied . a recent study showed that non-expert users can jailbreak LLMs by manipulating their prompts . |
| Approach: | They propose a formalism and a taxonomy of known (and possible) jailbreaks . they propose generating a dataset of model outputs across 3700 jailbreak prompts a 'prompt' attack is a new attack popularly categorized as "prompting injection attacks" |
| Outcome: | The proposed model exploits 3700 jailbreak prompts over 4 tasks to analyze their effectiveness . authors show that the model can learn to perform a new task on unseen examples . |
Fingerprinting LLMs via Prompt Injection (2026.acl-long)
Copied to clipboard
Yuepeng Hu, Zhengyuan Jiang, Mengyuan Li, Osama Ahmed, Zhicong Huang, Cheng Hong, Neil Zhenqiang Gong
| Challenge: | Existing provenance detection methods for large language models are infeasible for already published models and compare outputs using hand-crafted or random prompts. |
| Approach: | They propose a detection framework that constructs fingerprints by exploiting LLMs’ inherent vulnerability to prompt injection. |
| Outcome: | The proposed framework achieves high true positive rates while keeping false positive rates near zero. |
Distract Large Language Models for Automatic Jailbreak Attack (2024.emnlp-main)
Copied to clipboard
| Challenge: | Commercial large language models (LLMs) have made great progress in various NLP tasks. |
| Approach: | They propose a black-box jailbreak framework for automated red teaming of Large language models using an iterative optimization algorithm to conceal malicious content and memory reframing. |
| Outcome: | The proposed framework outperforms existing jailbreak defense methods and highlights the need to develop more effective and practical defense strategies. |
Jailbreak LLMs through Internal Stance Manipulation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to exploit LLMs' inherent safety mechanism, including GCG and AutoDAN, are ineffective for certain malicious requests. |
| Approach: | They propose a method that generates jailbreak prompts to suppress a refusal stance and induce affirmative responses by modifying adversarial prompts. |
| Outcome: | The proposed method outperforms the best baseline approach in Llama-2-7b-chat and achieves a 92.2% success rate across all models. |
What Does the Bot Say? Opportunities and Risks of Large Language Models in Social Media Bot Detection (2024.acl-long)
Copied to clipboard
| Challenge: | Social media bot detection has always been an arms race between advancements in machine learning and adversarial bot strategies to evade detection. |
| Approach: | They propose a mixture-of-heterogeneous-experts framework to divide and conquer diverse user information modalities and propose LLM-guided manipulation of user textual and structured information to evade detection. |
| Outcome: | The proposed framework outperforms state-of-the-art baselines on 1,000 annotated examples while bringing down existing detectors by 29.6% and harming calibration and reliability of bot detection systems. |