Unnatural language processing: How do language models handle machine-generated prompts? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language model prompt optimization research has shown that semantically and grammatically well-formed manually crafted prompts are outperformed by automatically generated token sequences with no apparent meaning or syntactic structure. |
| Approach: | They propose to use machine-generated prompts to probe how models respond to input that is not composed of natural language expressions. |
| Outcome: | The proposed model outperforms human-crafted prompts on a target zero-shot task. |
Similar Papers
Demystifying optimized prompts in language models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern language models (LMs) are not robust to out-of-distribution inputs. |
| Approach: | They investigate the composition of machine generated (“optimized”) prompts and the mechanisms by which LMs parse and build predictions from them. |
| Outcome: | The proposed prompts are primarily composed of punctuation and noun tokens, which are more rare in the training data. |
The language of prompting: What linguistic properties make a prompt successful? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that pretraining and instruction-tuned LLMs can achieve impressive performance on a multitude of tasks. |
| Approach: | They propose to use a standard for prompting research to better understand linguistic properties of LLMs. |
| Outcome: | The proposed standard would improve the performance of pre-trained and instruction-tuned LLMs on a multitude of tasks. |
Do Prompt-Based Models Really Understand the Meaning of Their Prompts? (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that prompts help models to learn faster in the same way that humans learn faster when provided with task instructions expressed in natural language. |
| Approach: | They experiment with 30 prompts manually written for natural language inference (NLI) they find that models can learn just as fast with many irrelevant or pathologically misleading prompts . |
| Outcome: | The proposed model can learn as fast with irrelevant or pathologically misleading prompts as with instructively “good” prompts. |
Prompting with Pseudo-Code Instructions (2023.emnlp-main)
Copied to clipboard
| Challenge: | ambiguity in natural language can hinder performance of large language models. |
| Approach: | They manually create a dataset of pseudo-code prompts for 132 different classification, QA, and generative language tasks, sourced from the Super-NaturalInstructions dataset. |
| Outcome: | The pseudo-code prompts improve the performance of two LLM families, BLOOM and CodeGen. |
Prompt2Model: Generating Deployable Models from Natural Language Instructions (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a step backward from traditional special-purpose NLP models . they require extensive computational resources for deployment and can be gated behind APIs . |
| Approach: | They propose a general-purpose method that takes a natural language task description and uses it to train a special-purpose model. |
| Outcome: | The proposed method outperforms a strong LLM by 20% while being 700 times smaller. |
Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality (2026.findings-acl)
Copied to clipboard
Qipeng Xie, Zi Liang, Jiafei Wu, Yufei Chen, Weizheng Wang, Wenao Ma, Zhong Ming, Haiqin Yang, Kaishun Wu
| Challenge: | Existing studies on prompt engineering have focused on optimizing models for performance under stylistic perturbations. |
| Approach: | They conduct the first analysis of n-gram token-level mechanisms . they find that higher average performance is inherently associated with lower variance and greater stability. |
| Outcome: | The proposed model reduces the variance of the generated code by 40% . the proposed model is based on a large-scale dataset of 132,000 prompt variants . |
Prompt Compression for Large Language Models: A Survey (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for improving LLM efficiency focus on optimizing the model itself, while prompt-centric methods focus on lowering the complexity of input. |
| Approach: | They propose to use prompt compression to optimize the compression encoder and combine hard and soft prompt methods to improve the efficiency of LLMs. |
| Outcome: | The proposed methods are categorized into hard prompt methods and soft prompt methods. |
Artificial Impressions: Evaluating Large Language Model Behavior Through the Lens of Trait Impressions (2025.emnlp-main)
Copied to clipboard
| Challenge: | We introduce and study artificial impressions–patterns in LLMs’ internal representations of prompts that resemble human impressions and stereotypes based on language. |
| Approach: | They introduce and study artificial impressions–patterns in LLMs’ internal representations of prompts that resemble human impressions and stereotypes based on language. |
| Outcome: | The proposed models predict impressions and model behavior based on the two-dimensional Stereotype Content Model (SCM). |
What’s in a prompt? Language models encode literary style in prompt embeddings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models encode textual information using high-dimensional latent spaces . many studies have investigated how conceptual content of words translates into geometrical relationships between their vector representations . |
| Approach: | They use literary pieces to show that intangible, rather than factual, aspects of the prompt are contained in deep representations. |
| Outcome: | The results show that word-to-vec(tor) embeddings are more complex than other models. |
Discourse-Aware Soft Prompting for Text Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in pre-trained langauge models (PLMs) have made great impact on text generation research. |
| Approach: | They propose to use hierarchical blocking to simulate a higher-level discourse structure of human written text and attention sparsity to learn sparse transformations on the softmax-function. |
| Outcome: | The proposed methods perform better on some generation tasks but don't generalize across all generation tasks. |