| Challenge: | In-context learning (ICL) heavily relies on selecting effective demonstrations to achieve outputs that better align with the expected results. |
| Approach: | They propose a method which integrates a demonstration validation perspective into this field and integrates it into the learning paradigm. |
| Outcome: | The proposed method surpasses all retrieval-based in-context learning techniques across both natural language understanding (NLU) and natural language generation (NLG) tasks. |
Similar Papers
Enhancing In-Context Learning via Implicit Demonstration Augmentation (2024.acl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a new paradigm for pre-trained language models that can make predictions for unseen inputs without updating parameters. |
| Approach: | They propose a method that enables a model to augmented copies of a demonstration by leveraging their deep feature distribution and a logit calibration mechanism. |
| Outcome: | The proposed method significantly improves the average and worst-case accuracy across diverse PLMs and tasks. |
Revisiting Demonstration Selection Strategies in In-Context Learning (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL). |
| Approach: | They propose a data- and model-dependent method to select models using in-context learning, TopK + ConE, and propose unified explanations for the effectiveness of previous methods. |
| Outcome: | The proposed method improves language understanding and generation tasks with different model scales. |
Rectifying Demonstration Shortcut in In-Context Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can solve tasks with a few demonstrations, but often rely on their pre-trained semantic priors rather than the input-label relationships to proceed with ICL prediction. |
| Approach: | They propose a demonstration-aware calibration method to improve LLMs' ability to learn new input-label relationships from demonstrations. |
| Outcome: | The proposed method improves the original ICL task and the task learning setting, and the results are generalized across three LLM families. |
MDR: Model-Specific Demonstration Retrieval at Inference Time for In-Context Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for retrieval-based in-context learning ignore model biases and fail to retrieve the most appropriate demonstrations for different LLMs. |
| Approach: | They propose a model-specific demonstration retrieval method that considers the biases of different LLMs at inference time. |
| Outcome: | The proposed method improves performance on seen and unseen tasks with multi-scale inference LLMs by up to 41.2%. |
Learning to Select In-Context Demonstration Preferred by Large Language Model (2025.findings-acl)
Copied to clipboard
| Challenge: | In-context learning (ICL) enables large language models to perform tasks with only a few examples as demonstrations. |
| Approach: | They propose a generative preference learning framework that leverages LLM feedback to directly optimize demonstration selection for ICL. |
| Outcome: | Experiments on 19 datasets across 11 task categories show that GenICL achieves superior performance than existing methods in selecting the most effective demonstrations. |
In-Context Learning with Iterative Demonstration Selection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing literature has highlighted the importance of selecting examples that are diverse or semantically similar to the test sample . Existing studies have shown that the optimal selection dimension, i.e., diversity or similarity, is task-specific. |
| Approach: | They propose to use zero-shot chain-of-thought reasoning to iteratively select examples that are diverse but still strongly correlated with the test sample as ICL demonstrations. |
| Outcome: | The proposed method outperforms existing demonstration selection methods on reasoning, question answering, and topic classification tasks. |
The Whole is Better than the Sum: Using Aggregated Demonstrations in In-Context Learning for Sequential Recommendation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown excellent performance on various NLP tasks. |
| Approach: | They propose a method that integrates multiple demonstration users into one aggregated demonstration to improve sequential recommendation. |
| Outcome: | The proposed method outperforms state-of-the-art LLM-based sequential recommendation methods on three recommendation datasets. |
Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Recent learning-based demonstration selection methods have proven beneficial to in-context learning (ICL) by choosing more useful exemplars. |
| Approach: | They propose two methods to capture task-agnostic similarities between input and output of LLMs. |
| Outcome: | The proposed methods integrate task-agnostic similarities of different levels between input and output of exemplars and test cases to eliminate costly data collection. |
Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods that rely on limited demos and out-of-demonstration (OOD) queries fail when faced with out- of-demotion queries. |
| Approach: | They propose a query-aware prompting method that elicits the inherent generalizability of large language models by query-based demo generation. |
| Outcome: | The proposed method outperforms state-of-the-art methods in the OOD setting and two public math benchmarks. |
Demonstration Augmentation for Zero-shot In-context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated an impressive capability known as In-context Learning (ICL), which enables them to acquire knowledge from textual demonstrations without the need for parameter updates. |
| Approach: | They propose to use model’s previously predicted historical samples as demonstrations for subsequent ones to improve model’ s performance. |
| Outcome: | The proposed method significantly outperforms the previous method and its predecessors in terms of inference cost and time. |