MDR: Model-Specific Demonstration Retrieval at Inference Time for In-Context Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for retrieval-based in-context learning ignore model biases and fail to retrieve the most appropriate demonstrations for different LLMs. |
| Approach: | They propose a model-specific demonstration retrieval method that considers the biases of different LLMs at inference time. |
| Outcome: | The proposed method improves performance on seen and unseen tasks with multi-scale inference LLMs by up to 41.2%. |
Similar Papers
Revisiting Demonstration Selection Strategies in In-Context Learning (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL). |
| Approach: | They propose a data- and model-dependent method to select models using in-context learning, TopK + ConE, and propose unified explanations for the effectiveness of previous methods. |
| Outcome: | The proposed method improves language understanding and generation tasks with different model scales. |
Strategic Demonstration Selection for Improved Fairness in LLM In-Context Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies highlight the effectiveness of using in-context learning (ICL) to steer large language models in processing tabular data. |
| Approach: | They propose a method that uses clustering and evolutionary strategies to curate a representative sample set from training data. |
| Outcome: | The proposed method significantly improves fairness across various metrics, showing its efficacy in real-world scenarios. |
D.Va: Validate Your Demonstration First Before You Use It (2025.acl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) heavily relies on selecting effective demonstrations to achieve outputs that better align with the expected results. |
| Approach: | They propose a method which integrates a demonstration validation perspective into this field and integrates it into the learning paradigm. |
| Outcome: | The proposed method surpasses all retrieval-based in-context learning techniques across both natural language understanding (NLU) and natural language generation (NLG) tasks. |
Learning to Select In-Context Demonstration Preferred by Large Language Model (2025.findings-acl)
Copied to clipboard
| Challenge: | In-context learning (ICL) enables large language models to perform tasks with only a few examples as demonstrations. |
| Approach: | They propose a generative preference learning framework that leverages LLM feedback to directly optimize demonstration selection for ICL. |
| Outcome: | Experiments on 19 datasets across 11 task categories show that GenICL achieves superior performance than existing methods in selecting the most effective demonstrations. |
Not All Demonstration Examples are Equally Beneficial: Reweighting Demonstration Examples for In-Context Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have recently gained the In-Context Learning ability . however, the quality of demonstration examples is usually uneven . |
| Approach: | They propose to determine optimal weights for demonstration examples and apply them during ICL. |
| Outcome: | The proposed approach outperforms conventional ICL on 8 classification tasks. |
In-Context Learning with Iterative Demonstration Selection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing literature has highlighted the importance of selecting examples that are diverse or semantically similar to the test sample . Existing studies have shown that the optimal selection dimension, i.e., diversity or similarity, is task-specific. |
| Approach: | They propose to use zero-shot chain-of-thought reasoning to iteratively select examples that are diverse but still strongly correlated with the test sample as ICL demonstrations. |
| Outcome: | The proposed method outperforms existing demonstration selection methods on reasoning, question answering, and topic classification tasks. |
Unified Demonstration Retriever for In-Context Learning (2023.acl-long)
Copied to clipboard
Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, Xipeng Qiu
| Challenge: | In-context learning is a new learning paradigm where a language model conditions on a few input-output pairs (demonstrations) and a test input, and directly outputs the prediction. |
| Approach: | They propose a single model to retrieve demonstrations for a wide range of tasks by combining training signals from various tasks into a unified list-wise ranking formulation by language model’s feedback. |
| Outcome: | The proposed model outperforms baselines on 30+ tasks across 13 task families and multiple data domains. |
In-Context Learning with Long-Context Models: An In-Depth Exploration (2025.naacl-long)
Copied to clipboard
| Challenge: | In-context learning is limited by context length, but it can be used for many tasks. |
| Approach: | They study the behavior of in-context learning at an extreme context length . example retrieval shows excellent performance at low context lengths but has diminished gains . |
| Outcome: | The proposed model can perform many tasks with reasonable accuracy when a few examples are provided in-context. |
Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Recent learning-based demonstration selection methods have proven beneficial to in-context learning (ICL) by choosing more useful exemplars. |
| Approach: | They propose two methods to capture task-agnostic similarities between input and output of LLMs. |
| Outcome: | The proposed methods integrate task-agnostic similarities of different levels between input and output of exemplars and test cases to eliminate costly data collection. |
Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on large-scale labeled support sets are not feasible in practical scenarios. |
| Approach: | They introduce a language model-based determinant point process that considers uncertainty and diversity of unlabeled instances for optimal selection. |
| Outcome: | The proposed method can effectively select canonical examples on 9 NLU and 2 Generation datasets. |