PromISe: Releasing the Capabilities of LLMs with Prompt Introspective Search (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks for large language models use uniform manual prompts, resulting in underestimation of performance. |
| Approach: | They propose a prompt introspective search framework that integrates self-introspect and self-refine to unlock the capabilities of LLMs. |
| Outcome: | The proposed framework significantly boosts the performance of 12 well-known LLMs compared to baseline methods. |
Similar Papers
PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are useful for low-resource scenarios and time-restricted applications. |
| Approach: | They propose a large-scale evaluation tool for large language models that uses prompts . they evaluate 720 prompt templates for open-source LLM-based metrics on MT and summarization datasets a 6.6M evaluations. |
| Outcome: | The proposed model evaluates 720 prompt templates on machine translation and summarization datasets. |
Heuristic-based Search Algorithm in Automatic Instruction-focused Prompt Optimization: A Survey (2025.findings-acl)
Copied to clipboard
Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley A. Malin, Sricharan Kumar
| Challenge: | Recent advances in Large Language Models (LLMs) have led to remarkable achievements across a variety of NLP tasks. |
| Approach: | They propose a taxonomy of automatic prompt optimization methods that explore and improve prompts with minimal human oversight. |
| Outcome: | The proposed methods can explore and improve prompts with minimal human oversight. |
PromptPrism: A Linguistically-Inspired Taxonomy for Prompts (2026.findings-eacl)
Copied to clipboard
| Challenge: | PromptPrism is a linguistically-inspired taxonomy that enables prompt analysis across three hierarchical levels. |
| Approach: | They propose a linguistically-inspired taxonomy that enables prompt analysis across three hierarchical levels: functional structure, semantic component, and syntactic pattern. |
| Outcome: | The proposed taxonomy bridges traditional language understanding with modern LLM research . it improves prompt quality and improves model performance across tasks . |
promptolution: A Unified, Modular Framework for Prompt Optimization (2026.eacl-demo)
Copied to clipboard
| Challenge: | Existing implementations of prompt optimization are tied to unmaintained, isolated codebases or require invasive integration into application frameworks. |
| Approach: | They propose a unified, modular open-source framework that integrates multiple contemporary discrete prompt optimizers within a single extensible system for both practitioners and researchers. |
| Outcome: | The proposed framework integrates multiple discrete prompt optimizers, supports systematic and reproducible benchmarking, and returns framework-agnostic prompt strings, enabling seamless integration into existing LLM pipelines while remaining agnosite to the underlying model implementation. |
SEE: Strategic Exploration and Exploitation for Cohesive In-Context Prompt Optimization (2025.acl-long)
Copied to clipboard
Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley A. Malin, Sricharan Kumar
| Challenge: | Existing approaches separate the optimization of prompt instructions and in-context learning examples, leading to incohesive, suboptimal results. |
| Approach: | They propose a framework that refines both prompt instructions and in-context learning examples. |
| Outcome: | The proposed framework outperforms state-of-the-art prompt optimization methods on 35 benchmark tasks. |
Entrospect: Information-Theoretic Self-Reflection Elicits Better Response Refinement of Small Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to self-reflection fail to deliver robust response refinement for models with parameter sizes of 10 billion or smaller. |
| Approach: | They propose to redesign Self-Refine and introduce an information-theoretic framework based on Chain-of-Thought prompt engineering to improve self-reflection in Small Language Models. |
| Outcome: | The proposed framework improves reasoning accuracy and computational efficiency by up to 36.2% under identical model and data settings. |
Enhancing Function-Calling Capabilities in LLMs: Strategies for Prompt Formats, Data Integration, and Multilingual Translation (2025.naacl-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) have significantly advanced autonomous agents, particularly in zero-shot tool usage, also known as function calling. |
| Approach: | They propose to integrate function descriptions into prompt formats and introduce a new Decision Token for conditional prompts. |
| Outcome: | The proposed decision token improves function-calling accuracy and relevance detection and a translation pipeline overcomes multilingual limitations. |
Prompt Injection in Automated Résumé Screening with Large Language Models: Single and Multi-Injection Settings (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used to screen and rank job applicants . a recent study shows that prompt injection can improve ranking when résumé quality is homogeneous . |
| Approach: | They study whether prompt injection distorts quality-based candidate selection . they find that it improves rankings when résumé quality is homogeneous . |
| Outcome: | The proposed model improves rankings when résumé quality is homogeneous and few candidates inject. |
HPSS: Heuristic Prompting Strategy Search for LLM Evaluators (2025.findings-acl)
Copied to clipboard
Bosi Wen, Pei Ke, Yufei Sun, Cunxiang Wang, Xiaotao Gu, Jinfeng Zhou, Jie Tang, Hongning Wang, Minlie Huang
| Challenge: | Existing efforts to optimize text evaluation prompts neglect the combinatorial impact of multiple factors, leading to insufficient optimization of the evaluation pipeline. |
| Approach: | They propose to integrate 8 key factors for evaluation prompts and integrate them into an algorithm that searches for well-behaved prompting strategies for LLM evaluators. |
| Outcome: | The proposed method outperforms existing methods and human-designed evaluation prompts on four evaluation tasks. |
Prompterator: Iterate Efficiently towards More Effective Prompts (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) use a process known as prompting to solve arbitrary language tasks. prompting is a non-trivial task that requires experimentation in order to arrive at a prompt that solves a specific task. |
| Approach: | They propose a tool that helps users iterate over different potential prompts and choose the best performing one based on human feedback. |
| Outcome: | The proposed tool is open source and easily extensible. |