An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels (2022.acl-long)
Copied to clipboard
Taylor Sorensen, Joshua Robinson, Christopher Rytting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mahmoud Khalil, Nancy Fulda, David Wingate
| Challenge: | Existing prompt engineering methods require labeled data and access to model parameters . a new method for selecting prompt templates without labeles and without direct access to the model is needed. |
| Approach: | They propose a method for selecting prompt templates without labeled examples and without direct access to the model. |
| Outcome: | The proposed method performs at almost oracle levels, without labels, on 7 datasets representing 7 different NLP tasks. |
Similar Papers
Prompt Engineering a Prompt Engineer (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies indicate that large language models can be meta-prompted to perform automatic prompt engineering, but their potential is limited due to insufficient guidance for complex reasoning in the meta-prompt. |
| Approach: | They propose to infuse three key components into a meta-prompt to guide reasoning . they find prompts that outperform “let’s think step by step” by 6.3% on MultiArith and 3.1% on GSM8K . |
| Outcome: | The proposed method outperforms “let’s think step by step” by 6.3% on MultiArith and 3.1% on GSM8K and outperfies baselines on counterfactual tasks by 6.9%. |
SOPL: A Sequential Optimal Learning Approach to Automated Prompt Engineering in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Using automated prompt engineering to identify effective features is essential for large language models. |
| Approach: | They propose an optimal learning framework for automated prompt engineering for black-box models . feature-based method is used to express prompt templates, which broadens the search space . |
| Outcome: | The proposed learning framework outperforms benchmark strategies on instruction induction tasks with limited budgets. |
What Makes Pre-trained Language Models Better Zero-shot Learners? (2023.acl-long)
Copied to clipboard
| Challenge: | Current methods for prompt learning in zero-shot scenarios rely on a development set with sufficient human-annotated data to select the best-performing prompt template. |
| Approach: | They propose a method for screening reasonable prompt templates in zero-shot text classification using language discrepancy. |
| Outcome: | The proposed method improves prediction performance in a realistic zero-shot setting, eliminating the need for labelled examples. |
Pre-trained Language Models Can be Fully Zero-Shot Learners (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to pre-trained language models require fine-tuning on labeled datasets or manually constructing proper prompts. |
| Approach: | They propose a nonparametric prompting PLM for fully zero-shot language understanding . they compare it to previous methods for text classification and text entailment . |
| Outcome: | The proposed method outperforms previous methods on diverse tasks. |
A Systematic Survey of Automatic Prompt Optimization Techniques (2025.emnlp-main)
Copied to clipboard
Kiran Ramnath, Kang Zhou, Sheng Guan, Soumya Smruti Mishra, Xuan Qi, Zhengyuan Shen, Shuai Wang, Sangmin Woo, Sullam Jeoung, Yawei Wang, Haozhu Wang, Han Ding, Yuzhe Lu, Zhichao Xu, Yun Zhou, Balasubramaniam Srinivasan, Qiaojing Yan, Yueyan Chen, Haibo Ding, Panpan Xu, Lin Lee Cheong
| Challenge: | Recent advances in prompt engineering have created impediments for end users to adopt . however, prompt engineering remains an impedance due to rapid advances in models, tasks, and associated best practices. |
| Approach: | They propose to define APO as a 5-part unifying framework and categorize all relevant works based on their salient features. |
| Outcome: | The proposed framework aims to improve the performance of large language models on various tasks. |
Self-Supervised Prompt Optimization (2025.findings-emnlp)
Copied to clipboard
Jinyu Xiang, Jiayi Zhang, Zhaoyang Yu, Xinbing Liang, Fengwei Teng, Jinhao Tu, Fashen Ren, Xiangru Tang, Sirui Hong, Chenglin Wu, Yuyu Luo
| Challenge: | Existing prompt optimization methods rely heavily on external references such as ground truth or by humans, limiting their applicability in real-world scenarios where such data is unavailable or costly to obtain. |
| Approach: | They propose a cost-efficient framework that discovers effective prompts for both closed and open-ended tasks without external reference. |
| Outcome: | The proposed framework outperforms state-of-the-art prompt optimization methods with significantly lower costs and fewer samples. |
Cold-Start Data Selection for Better Few-shot Language Model Fine-tuning: A Prompt-based Uncertainty Propagation Approach (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have achieved competitive performance with limited labeled data for many NLP tasks. |
| Approach: | They propose a prompt-based data selection method for pre-trained language models fine-tuning under cold-start scenarios. |
| Outcome: | The proposed method outperforms the strongest cold-start data selection baselines on six text classification datasets with 128 labels. |
GLaPE: Gold Label-agnostic Prompt Evaluation for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have explored leveraging the LLM itself as an optimizer to identify optimal prompts that maximize task accuracy. |
| Approach: | They propose a gold label-agnostic prompt evaluation method to reduce dependence on gold labels. |
| Outcome: | The proposed method produces more effective prompts even without gold labels. |
Do Prompt Positions Really Matter? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Prompt-based learning models have a high level of interest due to their ability to perform zero-shot and fewshot tasks. |
| Approach: | They conduct the most comprehensive analysis to date of prompt position for diverse natural language processing tasks. |
| Outcome: | The proposed model is more robust than previous models and is consistent even in instruction-tuned models. |
Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models demonstrate remarkable ability for learning to solve new tasks from a few examples. |
| Approach: | They propose to use templates to aggregate model predictions across multiple templates to improve model performance. |
| Outcome: | The proposed model ensembles boost model predictions while being robust to the choice of random set of templates. |