What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can perform in-context learning (ICL) with only a few demonstrations, but its mechanisms are not well-understood. |
| Approach: | They characterize two ways in which LLMs leverage demonstrations to solve tasks with a few demonstrations. |
| Outcome: | The proposed model achieves non-trivial performance with only TR, and TR does not improve with larger models or more demonstrations. |
Similar Papers
What Do Language Models Learn in Context? The Structured Task Hypothesis. (2024.acl-long)
Copied to clipboard
| Challenge: | Pre-trained large language models have exhibited an impressive ability to learn in context across various domains, e.g., code generation, education, medicine and even medicine. |
| Approach: | They taxonomize existing candidate theories into three competing hypotheses that explain LLMs’ ability to learn in context. |
| Outcome: | The proposed model can learn a task from in-context examples presented in a demonstration and generalize it to the prompt. |
Label Words as Local Task Vectors in In-Context Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable abilities, one of the most important being in-context learning (ICL). |
| Approach: | They hypothesized that the network creates a task vector in specific positions during ICL, which can be computed by averaging across the dataset. |
| Outcome: | The proposed model can achieve zero-shot performance with dummy inputs comparable to few-shot learning by patching the global task vector. |
In-Context Learning Creates Task Vectors (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a powerful new learning paradigm for Large Language Models (LLMs). |
| Approach: | They propose to use a model with a prompt and a query to learn a mapping based on two examples to produce the output. |
| Outcome: | The proposed model can learn functions from a simple structure based on a training set and a single task vector calculated from the training set. |
Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanism (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models exhibit remarkable in-context learning (ICL) capabilities, but the underlying working mechanism of ICL remains unclear. |
| Approach: | They propose a Two-Dimensional Coordinate System that unifies both views into a systematic framework that explains the behavior of ICL through two orthogonal variables: whether similar examples are presented in the demonstrations and whether LLMs can recognize the task. |
| Outcome: | The proposed method can interpret ICL for generation tasks effectively. |
The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis (2024.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a capability that enables large language models to excel in proficiency through demonstration examples. |
| Approach: | They present a survey on the interpretation and analysis of in-context learning . they focus on theoretical and empirical perspectives on the concept . |
| Outcome: | The proposed model can perform tasks with minimal examples without re-training and has demonstrated proficiency across various tasks with a minimal set of task-oriented examples. |
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022.emnlp-main)
Copied to clipboard
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer
| Challenge: | Large language models can in-context learn by conditioning on a few input-label pairs and making predictions for new inputs. |
| Approach: | They propose to use ground truth demonstrations to replace labels in demonstrations . they also show that other aspects of the demonstrations are key drivers of endtask performance . |
| Outcome: | The proposed model outperforms zeroshot inference on a wide range of tasks using ground truth demonstrations. |
Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel Tasks (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have transformed NLP with their remarkable In-context Learning capabilities. |
| Approach: | They propose to use large language models to generalize from labeled examples of predefined tasks to novel tasks . they use biological neurons and the Transformer architecture to study the potential for information sharing across tasks. |
| Outcome: | The proposed model can generalize from labeled examples of predefined tasks to novel tasks despite no examples from the target task in the context. |
Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on in-context learning mechanisms are not consistent . current research identifies two main approaches to explain the ICL mechanism . |
| Approach: | They propose a framework for evaluating in-context learning mechanisms by focusing on regression tasks. |
| Outcome: | The proposed framework can solve regression problems and then measure the extent to which the LLM retrieves its internal knowledge versus learning from in-context examples. |
Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Recent learning-based demonstration selection methods have proven beneficial to in-context learning (ICL) by choosing more useful exemplars. |
| Approach: | They propose two methods to capture task-agnostic similarities between input and output of LLMs. |
| Outcome: | The proposed methods integrate task-agnostic similarities of different levels between input and output of exemplars and test cases to eliminate costly data collection. |
Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a promising capability for large language models (LLMs) but its underlying mechanism remains unexplored. |
| Approach: | They propose a demonstration compression technique to expedite inference and an analysis framework for diagnosing ICL errors in GPT2-XL. |
| Outcome: | The proposed method improves ICL performance and expedites inference. |