Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022.emnlp-main)
Copied to clipboard
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer
| Challenge: | Large language models can in-context learn by conditioning on a few input-label pairs and making predictions for new inputs. |
| Approach: | They propose to use ground truth demonstrations to replace labels in demonstrations . they also show that other aspects of the demonstrations are key drivers of endtask performance . |
| Outcome: | The proposed model outperforms zeroshot inference on a wide range of tasks using ground truth demonstrations. |
Similar Papers
The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis (2024.findings-acl)
Copied to clipboard
Miaoran Zhang, Vagrant Gautam, Mingyang Wang, Jesujoba Alabi, Xiaoyu Shen, Dietrich Klakow, Marius Mosbach
| Challenge: | In-context learning is a popular inference strategy where large language models solve a task using only a few labeled demonstrations without updating the model parameters. |
| Approach: | They conduct multidimensional analysis of multilingual in-context learning using 5 models from different model families and 9 datasets covering classification and generation tasks. |
| Outcome: | The results show that demonstrations vary significantly across models, tasks, and languages. |
What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can perform in-context learning (ICL) with only a few demonstrations, but its mechanisms are not well-understood. |
| Approach: | They characterize two ways in which LLMs leverage demonstrations to solve tasks with a few demonstrations. |
| Outcome: | The proposed model achieves non-trivial performance with only TR, and TR does not improve with larger models or more demonstrations. |
Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations (2022.emnlp-main)
Copied to clipboard
Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, Taeuk Kim
| Challenge: | Intuitively, ground-truth labels should have as much impact in in-context learning as supervised learning, but the impact of the quality of demonstrations remains elusive. |
| Approach: | They propose to measure input-label correspondence and ground-truth label effect ratio . they propose to use verbosity of prompt templates and language model size as controlling factors . |
| Outcome: | The proposed metrics show that ground-truth labels have less impact than previously thought . the authors identify key components as controlling factors to achieve noise-resilient ICL . |
What Do Language Models Learn in Context? The Structured Task Hypothesis. (2024.acl-long)
Copied to clipboard
| Challenge: | Pre-trained large language models have exhibited an impressive ability to learn in context across various domains, e.g., code generation, education, medicine and even medicine. |
| Approach: | They taxonomize existing candidate theories into three competing hypotheses that explain LLMs’ ability to learn in context. |
| Outcome: | The proposed model can learn a task from in-context examples presented in a demonstration and generalize it to the prompt. |
Inference and Verbalization Functions During In-Context Learning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Previous work has found that, in some settings, ICL performance is minimally affected by using demonstrations with irrelevant label words. |
| Approach: | They hypothesize that large language models (LMs) perform in-context learning from a handful of demonstrations via two sequential processes: an inference function that solves the task and a verbalization function that maps the inferred answer to the label space. |
| Outcome: | The proposed model can be localized in specific layers across open-source models, including GEMMA-7B, MISTRAL-7B-V0.3, GEIMA-2-27B, and LLAMA-3.1-70B. |
Active Learning Principles for In-Context Learning with Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning has significantly enhanced predictive performance in few-shot learning settings. |
| Approach: | They propose to use pool-based Active Learning to identify the most informative demonstrations for few-shot learning over a single iteration to identify best demonstrations. |
| Outcome: | The proposed model outperforms all other methods, including random sampling, in the analysis of 24 classification and multi-choice tasks. |
Revisiting Demonstration Selection Strategies in In-Context Learning (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL). |
| Approach: | They propose a data- and model-dependent method to select models using in-context learning, TopK + ConE, and propose unified explanations for the effectiveness of previous methods. |
| Outcome: | The proposed method improves language understanding and generation tasks with different model scales. |
In-Context Learning with Long-Context Models: An In-Depth Exploration (2025.naacl-long)
Copied to clipboard
| Challenge: | In-context learning is limited by context length, but it can be used for many tasks. |
| Approach: | They study the behavior of in-context learning at an extreme context length . example retrieval shows excellent performance at low context lengths but has diminished gains . |
| Outcome: | The proposed model can perform many tasks with reasonable accuracy when a few examples are provided in-context. |
Enhancing In-Context Learning via Implicit Demonstration Augmentation (2024.acl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a new paradigm for pre-trained language models that can make predictions for unseen inputs without updating parameters. |
| Approach: | They propose a method that enables a model to augmented copies of a demonstration by leveraging their deep feature distribution and a logit calibration mechanism. |
| Outcome: | The proposed method significantly improves the average and worst-case accuracy across diverse PLMs and tasks. |
Active Example Selection for In-Context Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning performance is unstable across samples of examples, suggesting the idiosyncrasies of how language models acquire information. |
| Approach: | They propose a reinforcement learning algorithm for identifying generalizable policies to select demonstration examples and propose 'in-context learning' performance can be highly unstable across samples of examples, suggesting the idiosyncrasies of how language models acquire information. |
| Outcome: | The proposed model can perform tasks with examples with a 5.8% improvement on GPT-2 and GPT-3, but the improvement diminishes on larger models, suggesting emerging capabilities of large language models. |