| Challenge: | Instruction-tuning improves a model’s ability to learn in-context, but the mechanisms that drive in-constext learning are poorly understood. |
| Approach: | They propose to train a model on hundreds of tasks to improve its ability to learn in-context. |
| Outcome: | The proposed methods improve model transfer and in-context generalization, suggesting catastrophic forgetting may impact in-constext learning. |
Similar Papers
Exploring the Relationship between In-Context Learning and Instruction Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | In-Context Learning (ICL) and Instruction Tuning (IT) are two primary paradigms of adopting Large Language Models (LLMs) to downstream applications, but they are significantly different. |
| Approach: | They examine how the hidden states of Large Language Models change in these two paradigms by examining how they differ in implementation. |
| Outcome: | The proposed model changes the hidden states of LLMs as if its accompanying demonstrations were used to instructionally tune the model. |
Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained large language models retain task-specific knowledge, but where and to what extent they retain it remains unexplored. |
| Approach: | They investigate the task-specific information encoded in pre-trained LLMs and the effects of instruction tuning on their representations across over 60 NLP tasks. |
| Outcome: | The results show that pre-trained models retain task-specific knowledge . some tasks are already encoded in pre-train models, but others benefit from instruction tuning. |
Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Fine-tuning and in-context learning are two prevalent methods in imbuing large language models with task-specific knowledge. |
| Approach: | They propose to use a circuit shift theory to explain why in-context learning is superior to fine-tuning for tasks with implicit patterns. |
| Outcome: | The proposed method can grasp deep patterns and significantly improve accuracy on implicit patterns, compared with fine-tuning and in-context learning. |
Specialist or Generalist? Instruction Tuning for Specific NLP Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that instruction tuning can be a data-efficient method for transforming large language models into generalist models, but their performance lags behind specialist models trained exclusively for specific tasks. |
| Approach: | They propose to incorporate broadcoverage generalist instruction tuning into large language models to build a specialist model by incorporating task specificity and skill requirements. |
| Outcome: | The proposed method improves model performance when task coverage is broad and when training data is limited. |
How Does In-Context Learning Help Prompt Tuning? (2024.findings-eacl)
Copied to clipboard
| Challenge: | a growing number of parameter-efficient adaptation methods are needed to fine-tune large language models. |
| Approach: | They propose a method that combines prompt tuning and in-context learning to improve prompt tuning by concatenating a natural language demonstration with learned prompt embeddings. |
| Outcome: | The proposed method outperforms prompt tuning and prompt tuning on five language generation tasks. |
Large Language Models are Miscalibrated In-Context Learners (2025.findings-acl)
Copied to clipboard
| Challenge: | In-context Learning and Supervised Fine-Tuning have emerged as pre-dominant methodologies for machine learning and NLP. |
| Approach: | They propose to use self-ensembling to improve both performance and calibration of language models. |
| Outcome: | The proposed learning paradigms can achieve better calibration and better performance than the previous learning paradigm. |
Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning (2023.acl-short)
Copied to clipboard
| Challenge: | Recent studies on instruction tuning (IT) have achieved great performance with zero-shot generalizability to unseen tasks. |
| Approach: | They analyze how models utilize instructions during IT by comparing model training with altered vs. original instructions. |
| Outcome: | The proposed model outperforms naive models in low resource setting. |
How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes (2024.naacl-short)
Copied to clipboard
| Challenge: | Multi-task learning (MTL) for generalist models is a promising direction that offers transfer learning potential. |
| Approach: | They propose to combine multi-task learning (MTL) with in-context learning (ICL) to build models that can generalize to multiple tasks while being robust to out-of-distribution examples. |
| Outcome: | The proposed training strategies enable models to learn difficult tasks while mixing in prior tasks, denoted as mixed curriculum. |
Instruction Matters: A Simple yet Effective Task Selection for Optimized Instruction Tuning of Specific Tasks (2024.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that instruction tuning improves zero-shot generalization across various tasks and improves performance of specific tasks. |
| Approach: | They propose a task selection method that leverages instruction information alone to identify relevant tasks and optimize instruction tuning for specific tasks. |
| Outcome: | The proposed method is significantly more efficient than traditional approaches, which require complex measurements of pairwise transferability between tasks or the creation of data samples for the target task. |
Demystifying Instruction Mixing for Fine-tuning Large Language Models (2024.acl-srw)
Copied to clipboard
| Challenge: | Instruction tuning is effective for aligning large language models with human instructions, but the procedure to optimizing the mixing of instruction datasets is still unclear. |
| Approach: | They categorize instructions into three primary types: NLP downstream tasks, coding, and general chat. |
| Outcome: | The proposed method improves performance of large language models (LLMs) but it is difficult to combine different instruction datasets to optimize overall performance. |