Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale (2023.acl-long)
Copied to clipboard
| Challenge: | 70% of attention heads and 20% of the feed forward networks can be removed with minimal decline in task performance. |
| Approach: | They propose to investigate whether in-context learning is not uniform across all components of a large language model. |
| Outcome: | The proposed model can remove 70% of attention heads and 20% of feed forward networks with minimal decline in task performance. |
Similar Papers
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model (2022.naacl-main)
Copied to clipboard
Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Woo Ha, Nako Sung
| Challenge: | Recent studies on large-scale in-context language models have reported successful in-const zero- and few-shot learning ability. |
| Approach: | They investigate the effects of the pretraining corpus on in-context learning in a Korean-centric model. |
| Outcome: | The study shows that pretraining corpus size does not determine in-context learning ability . the findings suggest that in-constext learning is not always competitive . |
How do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads are Two Towers for Metric Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | In-context learning (ICL) is an emergent ability of large language models. |
| Approach: | They propose to use in-context learning to predict sentences with semantically-unrelated labels on 1% heads to investigate the mechanism. |
| Outcome: | The proposed methods reduce the majority label bias and recency bias by 22% and 17%, respectively. |
ICLEval: Evaluating In-Context Learning Ability of Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing evaluation frameworks focus on language abilities and knowledge, often overlooking the assessment of ICL ability. |
| Approach: | They propose to evaluate the ICL ability of Large Language Models (LLMs) using the ICLEval benchmark. |
| Outcome: | The proposed benchmark demonstrates that ICL ability is universally present in different LLMs and model size is not the sole determinant of ICL efficacy. |
What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can perform in-context learning (ICL) with only a few demonstrations, but its mechanisms are not well-understood. |
| Approach: | They characterize two ways in which LLMs leverage demonstrations to solve tasks with a few demonstrations. |
| Outcome: | The proposed model achieves non-trivial performance with only TR, and TR does not improve with larger models or more demonstrations. |
Are Emergent Abilities in Large Language Models just In-Context Learning? (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models have been claimed to acquire certain capabilities without having been specifically trained on them. |
| Approach: | They propose a theory that explains emergent abilities by taking into account their potential confounding factors and rigorously substantiate this theory through over 1000 experiments. |
| Outcome: | The proposed theory proves that emergent abilities are not truly emergental, but result from a combination of in-context learning, model memory, and linguistic knowledge. |
Emergent Abilities in Reduced-Scale Generative Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models can solve new tasks without task-specific fine-tuning. |
| Approach: | They propose to use pre-training data to pre-train 36 language models with billions of parameters to investigate whether emergent properties are tied to model size or can be demonstrated by smaller models. |
| Outcome: | The proposed model performs comparable to models trained on unrestricted language. |
MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Large-scale pre-trained vision-language models do not possess the ability to conduct in-context learning. |
| Approach: | They propose to meta-train a language model to perform in-context learning on NLP tasks and then transfer this model to VL tasks by attaching a visual encoder. |
| Outcome: | The proposed model outperforms the baseline model on VQA, OK-VQA, and GQA while having 20 times fewer parameters. |
One Task Vector is not Enough: A Large-Scale Study for In-Context Learning (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing studies limit comprehensive analysis of large language models based on task vectors . recent work points to "task vectors" as mechanism for encoding task rules . |
| Approach: | They propose a novel task vector with 30 input-output pairs for in-context learning . they use a few prompt-based examples to adapt to new tasks without weight updates . |
| Outcome: | Experiments with Llama-3-8B on QAF show task vector performance peaks at intermediate layer . complex tasks rely on multiple, subtask-specific vectors rather than a single vector . |
In-Context Learning with Long-Context Models: An In-Depth Exploration (2025.naacl-long)
Copied to clipboard
| Challenge: | In-context learning is limited by context length, but it can be used for many tasks. |
| Approach: | They study the behavior of in-context learning at an extreme context length . example retrieval shows excellent performance at low context lengths but has diminished gains . |
| Outcome: | The proposed model can perform many tasks with reasonable accuracy when a few examples are provided in-context. |
How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes (2024.naacl-short)
Copied to clipboard
| Challenge: | Multi-task learning (MTL) for generalist models is a promising direction that offers transfer learning potential. |
| Approach: | They propose to combine multi-task learning (MTL) with in-context learning (ICL) to build models that can generalize to multiple tasks while being robust to out-of-distribution examples. |
| Outcome: | The proposed training strategies enable models to learn difficult tasks while mixing in prior tasks, denoted as mixed curriculum. |