Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs (2024.emnlp-main)
Copied to clipboard
Krista Opsahl-Ong, Michael Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, Omar Khattab
| Challenge: | Language Model Programs (LMs) require crafting prompts that are jointly effective for all modules. |
| Approach: | They propose a novel algorithm for optimizing language model (LM) prompts for all modules by using program- and data-aware techniques and stochastic mini-batch evaluation functions. |
| Outcome: | The proposed algorithm outperforms baseline optimizers on five of seven diverse LM programs by as high as 13% accuracy. |
Similar Papers
Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows the potential of building more powerful Natural Language Processing systems by composing multiple skills of LMs into pipelines. |
| Approach: | They propose to combine weight and prompt optimization strategies to optimize a modular LM pipeline. |
| Outcome: | The proposed strategies outperform optimizing weights and prompts alone by 60% and 6% on average across LMs and tasks. |
MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators (2022.acl-long)
Copied to clipboard
| Challenge: | Prompting has been shown to be a promising approach for applying pre-trained language models to perform downstream tasks. |
| Approach: | They propose a method that divides the translation process into three stages using pre-trained language models. |
| Outcome: | The proposed method significantly improves translation performance of pre-trained language models on three translation tasks. |
LangProBe: a Language Program Benchmark (2025.findings-emnlp)
Copied to clipboard
Shangyin Tan, Lakshya A Agrawal, Arnav Singhvi, Liheng Lai, Michael J Ryan, Dan Klein, Omar Khattab, Koushik Sen, Matei Zaharia
| Challenge: | Composing language models into multi-step language programs is a mainstream paradigm for building AI systems, but tradeoffs in this space have only scarcely been studied before. |
| Approach: | They propose a benchmarking tool to evaluate the architectures and optimization strategies for language programs . they find that optimized language programs offer strong cost-quality Pareto improvement . |
| Outcome: | The proposed framework evaluates the impact of program architectures and optimizers on quality and cost. |
MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization (2023.findings-emnlp)
Copied to clipboard
Yuyan Chen, Zhihao Wen, Ge Fan, Zhengyu Chen, Wei Wu, Dayiheng Liu, Zhixu Li, Bang Liu, Yanghua Xiao
| Challenge: | Existing research emphasizes the importance of adapting prompts to specific tasks, rather than specific LLMs. |
| Approach: | They propose a model-adaptive prompt optimizer method that optimizes original prompts for each LLM in downstream tasks. |
| Outcome: | The proposed method can optimize prompts for an LLM in downstream tasks. |
FIPO: Free-form Instruction-oriented Prompt Optimization with Preference Dataset and Modular Fine-tuning Schema (2025.coling-main)
Copied to clipboard
| Challenge: | naive prompts can enhance the task performance of large language models, but they are resource-intensive. |
| Approach: | They propose an automatic prompt optimization method that refines naive prompts according to task outputs from in-box testing models. |
| Outcome: | The proposed method is based on a large-scale dataset and performed fairly across multiple models. |
Exploiting Edited Large Language Models as General Scientific Optimizers (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for solving optimization problems in scientific scenarios use observational feedback as additional textual descriptions, but these methods struggle to utilize it effectively. |
| Approach: | They propose a generalized approach to boost mathematical optimization in scientific scenarios by using observational feedback from LLMs as additional textual descriptions. |
| Outcome: | The proposed method outperforms existing state-of-the-art methods on six different tasks using six different LLM backbones. |
How Can We Know What Language Models Know? (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent work examines knowledge contained in language models by having the LM fill in the blanks of prompts such as “Obama is a __ by profession”. |
| Approach: | They propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. |
| Outcome: | The proposed methods improve accuracy from 31.1% to 39.6% on the LAMA benchmark for extracting relational knowledge from LMs. |
Learning from Contrastive Prompts: An Automated Prompt Optimization Framework (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing prompt optimization methods often underperform due to learning exclusively from incorrect samples. |
| Approach: | They propose a framework that leverages contrastive prompts to distinguish between high- and low-performing cases. |
| Outcome: | The proposed framework can generalize across open and proprietary models and NLU benchmarks. |
AutoDSPy: Automating Modular Prompt Design with Reinforcement Learning for Small and Large Language Models (2025.emnlp-industry)
Copied to clipboard
Nafew Azim, Abrar Ur Alam, Hasan Bin Omar, Abdullah Mohammad Muntasir Adnan Jami, Jawad Ibn Ahad, Muhammad Rafsan Kabir, Md. Ismail Hossain, Fuad Rahman, Mohammad Ruhul Amin, Shafin Rahman, Nabeel Mohammed
| Challenge: | Large Language Models excel at complex reasoning tasks, yet their performance hinges on the quality of their prompts and pipeline structures. |
| Approach: | They propose a framework that fully automates large language models' pipeline construction using reinforcement learning. |
| Outcome: | Experimental results show that autoDSPy outperforms DSPy benchmarks in accuracy gains and time. |
Modular and Parameter-Efficient Fine-Tuning for NLP Models (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | State-of-the-art language models in NLP perform best when fine-tuned even on small datasets. |
| Approach: | They provide an overview of parameter-efficient fine-tuning methods and highlight similarities and differences . they highlight benefits and usage scenarios of a neglected property of parameter efficient models . |
| Outcome: | This paper provides an overview of parameter-efficient fine-tuning methods . it highlights similarities and differences by presenting them in a unified view . |