PRompt Optimization in Multi-Step Tasks (PROMST): Integrating Human Feedback and Heuristic-based Sampling (2024.emnlp-main)
Copied to clipboard
| Challenge: | Prompt optimization aims to find the best prompt to a large language model (LLM) for a given task. |
| Approach: | They propose a method to optimize prompts for LLM-driven multi-step tasks using a human-designed feedback rule. |
| Outcome: | The proposed method outperforms human-engineered prompts and several other prompt optimization methods on 11 representative multi-step tasks. |
Similar Papers
PromptWizard: Optimizing Prompts via Task-Aware, Feedback-Driven Self-Evolution (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have transformed AI across diverse domains, with prompting being central to their success in guiding model outputs. |
| Approach: | They propose a framework for discrete prompt optimization that generates human-readable prompts using feedback-driven critique and synthesis process. |
| Outcome: | The proposed framework improves prompt quality across 45 tasks and reduces API calls, token usage and overall cost. |
Heuristic-based Search Algorithm in Automatic Instruction-focused Prompt Optimization: A Survey (2025.findings-acl)
Copied to clipboard
Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley A. Malin, Sricharan Kumar
| Challenge: | Recent advances in Large Language Models (LLMs) have led to remarkable achievements across a variety of NLP tasks. |
| Approach: | They propose a taxonomy of automatic prompt optimization methods that explore and improve prompts with minimal human oversight. |
| Outcome: | The proposed methods can explore and improve prompts with minimal human oversight. |
Prompterator: Iterate Efficiently towards More Effective Prompts (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) use a process known as prompting to solve arbitrary language tasks. prompting is a non-trivial task that requires experimentation in order to arrive at a prompt that solves a specific task. |
| Approach: | They propose a tool that helps users iterate over different potential prompts and choose the best performing one based on human feedback. |
| Outcome: | The proposed tool is open source and easily extensible. |
iPrOp: Interactive Prompt Optimization for Large Language Models with a Human in the Loop (2025.acl-srw)
Copied to clipboard
| Challenge: | Prompt engineering has made significant contributions to the era of large language models, yet its effectiveness depends on the skills of a prompt author. |
| Approach: | They propose a novel approach to prompt optimization that bridges manual prompt engineering and automatic prompt optimization by providing task-specific guidance. |
| Outcome: | The proposed approach bridges manual prompt engineering and automatic prompt optimization while offering users the flexibility to assess evolving prompts. |
promptolution: A Unified, Modular Framework for Prompt Optimization (2026.eacl-demo)
Copied to clipboard
| Challenge: | Existing implementations of prompt optimization are tied to unmaintained, isolated codebases or require invasive integration into application frameworks. |
| Approach: | They propose a unified, modular open-source framework that integrates multiple contemporary discrete prompt optimizers within a single extensible system for both practitioners and researchers. |
| Outcome: | The proposed framework integrates multiple discrete prompt optimizers, supports systematic and reproducible benchmarking, and returns framework-agnostic prompt strings, enabling seamless integration into existing LLM pipelines while remaining agnosite to the underlying model implementation. |
A Systematic Survey of Automatic Prompt Optimization Techniques (2025.emnlp-main)
Copied to clipboard
Kiran Ramnath, Kang Zhou, Sheng Guan, Soumya Smruti Mishra, Xuan Qi, Zhengyuan Shen, Shuai Wang, Sangmin Woo, Sullam Jeoung, Yawei Wang, Haozhu Wang, Han Ding, Yuzhe Lu, Zhichao Xu, Yun Zhou, Balasubramaniam Srinivasan, Qiaojing Yan, Yueyan Chen, Haibo Ding, Panpan Xu, Lin Lee Cheong
| Challenge: | Recent advances in prompt engineering have created impediments for end users to adopt . however, prompt engineering remains an impedance due to rapid advances in models, tasks, and associated best practices. |
| Approach: | They propose to define APO as a 5-part unifying framework and categorize all relevant works based on their salient features. |
| Outcome: | The proposed framework aims to improve the performance of large language models on various tasks. |
HPSS: Heuristic Prompting Strategy Search for LLM Evaluators (2025.findings-acl)
Copied to clipboard
Bosi Wen, Pei Ke, Yufei Sun, Cunxiang Wang, Xiaotao Gu, Jinfeng Zhou, Jie Tang, Hongning Wang, Minlie Huang
| Challenge: | Existing efforts to optimize text evaluation prompts neglect the combinatorial impact of multiple factors, leading to insufficient optimization of the evaluation pipeline. |
| Approach: | They propose to integrate 8 key factors for evaluation prompts and integrate them into an algorithm that searches for well-behaved prompting strategies for LLM evaluators. |
| Outcome: | The proposed method outperforms existing methods and human-designed evaluation prompts on four evaluation tasks. |
Local Prompt Optimization (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing prompt optimization methods optimize prompts globally, but they lack the correct words for a task. |
| Approach: | They propose a local prompt optimization process that integrates with any general automatic prompt engineering method to optimize a prompt over a large vocabulary. |
| Outcome: | The proposed method improves on Math Reasoning and BIG-bench Hard benchmarks and shows that it can converge to the optimal prompt faster than global methods. |
P3: Prompts Promote Prompting (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in prompt optimization have shown effectiveness of using multiple components to optimize models . however, such unilateral approaches often yield suboptimal results due to interdependent nature of these components. |
| Approach: | They propose a self-improvement framework that optimizes both system and user prompts . they use offline optimized prompts to promote online prompt optimization . |
| Outcome: | The proposed framework improves performance on general and reasoning tasks. |
Bandit-Based Prompt Design Strategy Selection Improves Prompt Optimizers (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing prompt optimization methods have found effective prompts, but they often differ from sophisticated prompts carefully designed by human experts. |
| Approach: | They propose to integrate prompt design strategies into prompt optimization by using a Thompson sampling-based approach. |
| Outcome: | The proposed method incorporates prompt design strategies into the prompt optimization process. |