Strings from the Library of Babel: Random Sampling as a Strong Baseline for Prompt Optimisation (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent prompt optimisation approaches use the generative nature of language models to produce prompts, even rivaling the performance of human-curated prompts. |
| Approach: | They propose to randomly sample tokens from the model vocabulary as "separators" they show that random separators are competitive baselines, having less than a 1% difference compared to previous self-optimisation methods. |
| Outcome: | The proposed method outperforms human-curated prompts in nine text classification tasks and eight language models and has a 40% chance of performing better than human-generated separators. |
Similar Papers
Prompterator: Iterate Efficiently towards More Effective Prompts (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) use a process known as prompting to solve arbitrary language tasks. prompting is a non-trivial task that requires experimentation in order to arrive at a prompt that solves a specific task. |
| Approach: | They propose a tool that helps users iterate over different potential prompts and choose the best performing one based on human feedback. |
| Outcome: | The proposed tool is open source and easily extensible. |
Demystifying optimized prompts in language models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern language models (LMs) are not robust to out-of-distribution inputs. |
| Approach: | They investigate the composition of machine generated (“optimized”) prompts and the mechanisms by which LMs parse and build predictions from them. |
| Outcome: | The proposed prompts are primarily composed of punctuation and noun tokens, which are more rare in the training data. |
CRL-Prompt: Contrastive and Reinforcement Learning for Soft Prompt Tuning for Text Classification (2026.acl-srw)
Copied to clipboard
| Challenge: | Manual prompt engineering is time-consuming, non-scalable, and brittle, while current auto-prompting techniques are far from maturity. |
| Approach: | They propose a two-stage method for prompt learning of frozen language models, CRL-Prompt, based on soft prompt initialization followed by contrastive and reinforcement-based refinement. |
| Outcome: | The proposed method achieves consistent improvements over baseline prompt tuning strategies, with gains of up to 2.2% while training fewer than 0.25% of model parameters. |
PRompt Optimization in Multi-Step Tasks (PROMST): Integrating Human Feedback and Heuristic-based Sampling (2024.emnlp-main)
Copied to clipboard
| Challenge: | Prompt optimization aims to find the best prompt to a large language model (LLM) for a given task. |
| Approach: | They propose a method to optimize prompts for LLM-driven multi-step tasks using a human-designed feedback rule. |
| Outcome: | The proposed method outperforms human-engineered prompts and several other prompt optimization methods on 11 representative multi-step tasks. |
Distribution Prompting: Understanding the Expressivity of Language Models Through the Next-Token Distributions They Can Produce (2025.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive neural language models (LMs) generate a probability distribution over tokens at each time step given a prompt. |
| Approach: | They propose to find a prompt that induces LMs to output a distribution as close as possible to the target, using either soft or hard gradient-based prompt tuning. |
| Outcome: | The proposed model is able to generate a distribution as close as possible to a target given a prompt, and it can be used to approximate distributions with low or high entropy. |
Prompt Compression for Large Language Models: A Survey (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for improving LLM efficiency focus on optimizing the model itself, while prompt-centric methods focus on lowering the complexity of input. |
| Approach: | They propose to use prompt compression to optimize the compression encoder and combine hard and soft prompt methods to improve the efficiency of LLMs. |
| Outcome: | The proposed methods are categorized into hard prompt methods and soft prompt methods. |
Learning from Contrastive Prompts: An Automated Prompt Optimization Framework (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing prompt optimization methods often underperform due to learning exclusively from incorrect samples. |
| Approach: | They propose a framework that leverages contrastive prompts to distinguish between high- and low-performing cases. |
| Outcome: | The proposed framework can generalize across open and proprietary models and NLU benchmarks. |
The Power of Scale for Parameter-Efficient Prompt Tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Unlike discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. |
| Approach: | They propose a mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. |
| Outcome: | The proposed method outperforms fewshot learning using GPT-3 and matches the quality of model tuning as models exceed billions of parameters. |
Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality (2026.findings-acl)
Copied to clipboard
Qipeng Xie, Zi Liang, Jiafei Wu, Yufei Chen, Weizheng Wang, Wenao Ma, Zhong Ming, Haiqin Yang, Kaishun Wu
| Challenge: | Existing studies on prompt engineering have focused on optimizing models for performance under stylistic perturbations. |
| Approach: | They conduct the first analysis of n-gram token-level mechanisms . they find that higher average performance is inherently associated with lower variance and greater stability. |
| Outcome: | The proposed model reduces the variance of the generated code by 40% . the proposed model is based on a large-scale dataset of 132,000 prompt variants . |
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts (2021.naacl-main)
Copied to clipboard
| Challenge: | Pretrained language models retain factual knowledge that can be extracted with a sentential prompt. |
| Approach: | They propose to learn prompts by gradient descent, either fine-tuning prompts or starting from random initialization. |
| Outcome: | The proposed approach outperforms existing methods on English LMs and tasks. |