Unlocking Black-Box Prompt Tuning Efficiency via Zeroth-Order Optimization

Challenge:	Prompt optimization is an important technique for adapting Large Language Models (LLMs) to specific tasks.
Approach:	They propose a zeroth-order approach which enables efficient prompt tuning solely via inference APIs.
Outcome:	The proposed approach outperforms existing black-box prompt tuning methods in terms of performance and convergence speed.

Similar Papers

Overview

Challenge

Approach

Outcome

Make Prompt-based Black-Box Tuning Colorful: Boosting Model Generalization from Three Orthogonal Perspectives (2024.lrec-main)

Copied to clipboard

Qiushi Sun, Chengcheng Han, Nuo Chen, Renyu Zhu, Jingyang Gong, Xiang Li, Ming Gao

Language Models

Challenge:	Large language models (LLMs) have shown increasing power on NLP tasks. however, tuning these models for downstream tasks usually requires exorbitant costs.
Approach:	They propose a black-box tuning technique that optimizes task-specific prompts without accessing gradients and hidden representations.
Outcome:	The proposed method improves performance under few-shot learning scenarios.

Hi-ZFO: Hierarchical Zeroth- and First-Order LLM Fine-Tuning via Importance-Guided Tensor Selection (2026.findings-acl)

Copied to clipboard

Feihu Jin, Ying Tan

Language Models

Challenge:	generative tasks require a high degree of exploratory capacity, but zeroth-order methods suffer from slow convergence . generative task-specific methods tend to converge toward local minima, causing noise and inefficient estimation .
Approach:	They propose a framework that synergizes FO precision with exploratory capability of ZO estimation.
Outcome:	The proposed framework synergizes precision of FO gradients with exploratory capability of ZO estimation.

Reliable Gradient-free and Likelihood-free Prompt Tuning (2023.findings-eacl)

Copied to clipboard

Maohao Shen, Soumya Ghosh, Prasanna Sattigeri, Subhro Das, Yuheng Bu, Gregory Wornell

Language Models

Challenge:	Large pre-trained language models are often offered as black-box APIs due to privacy or commercial constraints.
Approach:	They propose to tune the soft prompts without requiring gradient computation and extend the model to include a distribution over prompts.
Outcome:	The proposed methods are competitive with gradient-based approaches with full access to the PLM.

ZERA: Zero-init Instruction Evolving Refinement Agent – From Zero Instructions to Structured Prompts via Principle-based Optimization (2025.emnlp-main)

Copied to clipboard

Seungyoun Yi, Minsoo Khang, Sungrae Park

Low-Resource NLP

Challenge:	Existing methods to improve large language model performance focus on user prompts and require large sample sizes and long iteration cycles.
Approach:	They propose a framework that jointly optimizes both system and user prompts . they evaluate ZERA across five LLMs and nine diverse datasets spanning reasoning, summarization, and code generation tasks.
Outcome:	The proposed framework improves prompt construction over baselines and is available on github . it scores prompts using eight generalizable criteria and revises prompts based on structured critiques.

XPrompt: Exploring the Extreme of Prompt Tuning (2022.emnlp-main)

Copied to clipboard

Fang Ma, Chen Zhang, Lei Ren, Jingang Wang, Qifan Wang, Wei Wu, Xiaojun Quan, Dawei Song

Responsible & Trustworthy NLP

Challenge:	Prompt tuning learns soft prompts to condition pre-trained Language Models for performing downstream tasks in a parameter-efficient manner.
Approach:	They propose a Prompt tuning model with an eXtremely small scale that learns soft prompts to condition the frozen Pre-trained Language Models for performing downstream tasks in a parameter-efficient manner.
Outcome:	The proposed model outperforms the vanilla Prompt-Tuning and can significantly improve across tasks and model scales.

Hard Prompts Made Interpretable: Sparse Entropy Regularization for Prompt Tuning with RL (2024.acl-long)

Copied to clipboard

Yunseon Choi, Sangmin Bae, Seonghyun Ban, Minchan Jeong, Chuheng Zhang, Lei Song, Li Zhao, Jiang Bian, Kee-Eung Kim

Responsible & Trustworthy NLP

Challenge:	Prompt tuning is an important technique for directing model behaviors and eliciting desired responses.
Approach:	They propose to find optimal prompt tokens using soft Q-learning to optimize models for prompt tuning.
Outcome:	The proposed method improves on baseline prompt tuning, and the results are more natural and interpretable.

Black-Box Prompt Optimization: Aligning Large Language Models without Model Training (2024.acl-long)

Copied to clipboard

Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, Minlie Huang

Language Models

Challenge:	Large language models are often not well aligned with human intents, which requires additional training.
Approach:	They propose to use Black-Box Prompt Optimization (BPO) to perform alignments on large language models that are not well aligned with human intents.
Outcome:	The proposed model outperforms existing models and is model-agnostic.

Ultra-Low-Dimensional Prompt Tuning via Random Projection (2026.eacl-long)

Copied to clipboard

Zijun Wu, Yongchang Hao, Lili Mou

Semantic Text Processing

Challenge:	Prompt tuning addresses parameter-efficiency by learning embeddings, but these embeddements are typically tied to the model’s hidden dimensionality, limiting parameter saving.
Approach:	They propose a parameter-efficient method that learns prompt embeddings exclusively in the input layer of the model and uses a frozen random matrix for up-projection.
Outcome:	The proposed method outperforms previous methods using significantly fewer parameters while maintaining performance.

DLPO: Towards a Robust, Efficient, and Generalizable Prompt Optimization Framework from a Deep-Learning Perspective (2025.findings-emnlp)

Copied to clipboard

Dengyun Peng, Yuhang Zhou, Qiguang Chen, JinHao Liu, Jingjing Chen, Libo Qin, Wanxiang Che

Responsible & Trustworthy NLP

Challenge:	Existing methods for prompt optimization still face challenges in robustness, efficiency, and generalization.
Approach:	They propose 7 new approaches inspired by traditional deep learning paradigms for prompt optimization that integrate text-based gradient optimization.
Outcome:	The proposed methods integrate deep learning paradigms into text-based gradient optimization.

The Power of Scale for Parameter-Efficient Prompt Tuning (2021.emnlp-main)

Copied to clipboard

Brian Lester, Rami Al-Rfou, Noah Constant

Responsible & Trustworthy NLP

Challenge:	Unlike discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples.
Approach:	They propose a mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks.
Outcome:	The proposed method outperforms fewshot learning using GPT-3 and matches the quality of model tuning as models exceed billions of parameters.