Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods that rely on limited demos and out-of-demonstration (OOD) queries fail when faced with out- of-demotion queries. |
| Approach: | They propose a query-aware prompting method that elicits the inherent generalizability of large language models by query-based demo generation. |
| Outcome: | The proposed method outperforms state-of-the-art methods in the OOD setting and two public math benchmarks. |
Similar Papers
Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown striking ability to adapt to target tasks with a few input-output demonstrations. |
| Approach: | They propose a framework which bootstraps LMs’ intrinsic capabilities to perform zero-shot ICL. |
| Outcome: | The proposed framework outperforms baselines on 23 BIG-Bench Hard tasks on average accuracy and head-to-head comparison. |
D.Va: Validate Your Demonstration First Before You Use It (2025.acl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) heavily relies on selecting effective demonstrations to achieve outputs that better align with the expected results. |
| Approach: | They propose a method which integrates a demonstration validation perspective into this field and integrates it into the learning paradigm. |
| Outcome: | The proposed method surpasses all retrieval-based in-context learning techniques across both natural language understanding (NLU) and natural language generation (NLG) tasks. |
Just Adjust One Prompt: Enhancing In-Context Dialogue Scoring via Constructing the Optimal Subgraph of Demonstrations and Prompts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using large language models as chatbots can cause hallucinations and lack of empathy, authors report . a dimension-agnostic scoring method is proposed to improve the performance of chatbot performance . |
| Approach: | They propose a dimension-agnostic scoring method that leverages in-context learning . they propose to automatically generate prompts and then request the LLM multiple times . |
| Outcome: | The proposed method outperforms baselines on five datasets. |
Demonstration Augmentation for Zero-shot In-context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated an impressive capability known as In-context Learning (ICL), which enables them to acquire knowledge from textual demonstrations without the need for parameter updates. |
| Approach: | They propose to use model’s previously predicted historical samples as demonstrations for subsequent ones to improve model’ s performance. |
| Outcome: | The proposed method significantly outperforms the previous method and its predecessors in terms of inference cost and time. |
Self-Prompting Large Language Models for Zero-Shot Open-Domain QA (2024.naacl-long)
Copied to clipboard
| Challenge: | Open-Domain Question Answering (ODQA) aims to answer questions without explicitly providing specific background documents. |
| Approach: | They propose a framework to explicitly utilize the massive knowledge encoded in LLM parameters and their strong instruction understanding abilities. |
| Outcome: | The proposed framework surpasses state-of-the-art methods on three widely-used ODQA datasets and achieves comparable performance with customized fine-tuned models on full training data. |
Navigating the Unknown: Intent Classification and Out-of-Distribution Detection Using Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Out-of-Distribution (OOD) detection requires great generalization capability . |
| Approach: | They propose a method that is cost-efficient, high-performing, highly robust and versatile enough to be used with smaller LLMs without sacrificing performance. |
| Outcome: | The proposed method is cost-efficient, high-performing, robust, and versatile enough to be used with smaller LLMs without sacrificing performance. |
PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are useful for low-resource scenarios and time-restricted applications. |
| Approach: | They propose a large-scale evaluation tool for large language models that uses prompts . they evaluate 720 prompt templates for open-source LLM-based metrics on MT and summarization datasets a 6.6M evaluations. |
| Outcome: | The proposed model evaluates 720 prompt templates on machine translation and summarization datasets. |
Unleashing the Power of Large Language Models in Zero-shot Relation Extraction via Self-Prompting (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for zero-shot Relation Extraction (RE) lack detailed, context-specific prompts for understanding various sentences and relations. |
| Approach: | They propose a framework that uses a three-stage diversity approach to prompt LLMs by generating multiple synthetic samples that encapsulate specific relations from scratch. |
| Outcome: | The proposed framework outperforms existing LLM-based zero-shot RE methods on benchmark datasets and shows that it produces high-quality synthetic data that enhances performance. |
More Samples or More Prompts? Exploring Effective Few-Shot In-Context Learning for LLMs with In-Context Sampling (2024.findings-naacl)
Copied to clipboard
Bingsheng Yao, Guiming Chen, Ruishi Zou, Yuxuan Lu, Jiachen Li, Shao Zhang, Yisi Sang, Sijia Liu, James Hendler, Dakuo Wang
| Challenge: | Existing studies on LLM prompting focus on selecting a better set of data samples inside one single prompt input, but why not design and leverage multiple ICL prompts together to further improve the LLM’s performance? |
| Approach: | They propose a low-resource LLM prompting technique to optimize the construction of multiple ICL prompt inputs to produce confident predictions. |
| Outcome: | The proposed technique can produce confident predictions by optimizing the construction of multiple ICL prompt inputs on four NLI datasets and one QA dataset. |
Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies (2023.findings-emnlp)
Copied to clipboard
Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, Dragomir Radev
| Challenge: | In-context learning (ICL) is a new approach to natural language processing tasks that rely on large language models to make predictions based on context . recent studies have shown that neural symbolic design is the preferred choice for question answering systems because of its limited working memory and unreliable long-term memory. |
| Approach: | They propose to extend in-context learning to question answering tasks that utilize structured knowledge sources and to explore various prompt design strategies for employing LLMs. |
| Outcome: | The proposed approach outperforms the state-of-the-art system by 2.5 points and the best fine-tuned system by 5.1 points on the Spider dataset. |