| Challenge: | We find that many natural-language prompts can be replaced by corresponding unintelligible prompts that provably elicit similar behavior in language models. |
| Approach: | They find that natural-language prompts can be replaced by corresponding unintelligible prompts that elicit similar behavior in language models. |
| Outcome: | The proposed prompts are obfuscated and uninterpretable but mimic the original natural-language prompts . the problem has applications of independent interest, the authors argue . |
Similar Papers
Do Prompt-Based Models Really Understand the Meaning of Their Prompts? (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that prompts help models to learn faster in the same way that humans learn faster when provided with task instructions expressed in natural language. |
| Approach: | They experiment with 30 prompts manually written for natural language inference (NLI) they find that models can learn just as fast with many irrelevant or pathologically misleading prompts . |
| Outcome: | The proposed model can learn as fast with irrelevant or pathologically misleading prompts as with instructively “good” prompts. |
Unnatural language processing: How do language models handle machine-generated prompts? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language model prompt optimization research has shown that semantically and grammatically well-formed manually crafted prompts are outperformed by automatically generated token sequences with no apparent meaning or syntactic structure. |
| Approach: | They propose to use machine-generated prompts to probe how models respond to input that is not composed of natural language expressions. |
| Outcome: | The proposed model outperforms human-crafted prompts on a target zero-shot task. |
The Threat of PROMPTS in Large Language Models: A System and User Prompt Perspective (2025.findings-acl)
Copied to clipboard
| Challenge: | Prompts are essential for guiding model output and influencing content generation. |
| Approach: | They propose to attack models with prompt leakage and prompt jailbreak attacks . they summarize the experimental setups of these methods and explore the relationship between prompt threats and prompt injection attacks. |
| Outcome: | The proposed methods summarize the experimental setups and examine the relationship between prompt threats and prompt injection attacks. |
Are Language Models Worse than Humans at Following Prompts? It’s Complicated (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work shows that language models perform surprisingly well when given intentionally irrelevant or misleading prompts. |
| Approach: | They challenge an assumption that humans would perform badly when given pathological instructions by ignoring irrelevant prompts and following them faithfully when given misleading instructions. |
| Outcome: | The proposed model performs well when given intentionally irrelevant or misleading prompts, whereas models do not. |
Demystifying optimized prompts in language models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern language models (LMs) are not robust to out-of-distribution inputs. |
| Approach: | They investigate the composition of machine generated (“optimized”) prompts and the mechanisms by which LMs parse and build predictions from them. |
| Outcome: | The proposed prompts are primarily composed of punctuation and noun tokens, which are more rare in the training data. |
Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | a high prompt sensitivity has been widely accepted as a core limitation of large language models . a recent study suggests that prompt senescence may be an artifact of evaluation processes . |
| Approach: | They examine whether prompt sensitivity is an inherent weakness or an artifact of evaluation . they find that heuristic evaluation methods overlook semantically correct responses . large language models have achieved remarkable success across a wide range of tasks . |
| Outcome: | The proposed model is more robust to prompt templates than previously thought . the authors show that prompt sensitivity may be an artifact of evaluation rather than a flaw . |
How Important is ‘Perfect’ English for Machine Translation Prompts? (2026.findings-eacl)
Copied to clipboard
Patrícia Schmidtová, Niyati Bafna, Seth Aycock, Gianluca Vico, Wiktor Kamzela, Kathy Hämmerl, Vilém Zouhar
| Challenge: | Large language models (LLMs) are largely trained on and respond best to English prompts, but are also sensitive to errors in user prompts. |
| Approach: | They propose to model a range of error types exhibited by second language English speakers and quantify their impact on LLM performance. |
| Outcome: | The proposed model is brittle to natural spelling errors but not to errors at the phrasal level, but the variance in quality caused by these errors is lower than the variance over the initial prompt choice. |
Extracting Prompts by Inverting LLM Outputs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Unlike previous methods, output2prompt only needs outputs of normal user queries. |
| Approach: | They propose a black-box method that extracts the model's prompt without accessing its logits and without adversarial or jailbreaking queries. |
| Outcome: | The proposed method extracts the prompt that generated the outputs without accessing the model's logits and without adversarial or jailbreaking queries. |
Demystifying Prompts in Language Models via Perplexity Estimation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models can be prompted to perform a wide variety of tasks with zero- and few-shot learning. |
| Approach: | They propose a method to automatically extend a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. |
| Outcome: | The proposed method extends a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. |
MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts? (2025.findings-emnlp)
Copied to clipboard
Muntasir Wahed, Xiaona Zhou, Kiet A. Nguyen, Tianjiao Yu, Nirav Diwan, Gang Wang, Dilek Hakkani-Tür, Ismini Lourentzou
| Challenge: | Recent advances in Large Language Models have significantly enhanced their code generation capabilities, but their robustness against adversarial misuse remains underexplored. |
| Approach: | They introduce a code decomposition attack where a malicious coding task is broken down into subtasks across multiple conversational turns to evade safety filters. |
| Outcome: | The proposed code decomposition attacks exploits multi-turn malicious coding prompts . the proposed model improves rejection rates while preserving coding ability . |