How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes (2024.naacl-short)
Copied to clipboard
| Challenge: | Multi-task learning (MTL) for generalist models is a promising direction that offers transfer learning potential. |
| Approach: | They propose to combine multi-task learning (MTL) with in-context learning (ICL) to build models that can generalize to multiple tasks while being robust to out-of-distribution examples. |
| Outcome: | The proposed training strategies enable models to learn difficult tasks while mixing in prior tasks, denoted as mixed curriculum. |
Similar Papers
In-Context Learning Creates Task Vectors (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a powerful new learning paradigm for Large Language Models (LLMs). |
| Approach: | They propose to use a model with a prompt and a query to learn a mapping based on two examples to produce the output. |
| Outcome: | The proposed model can learn functions from a simple structure based on a training set and a single task vector calculated from the training set. |
What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can perform in-context learning (ICL) with only a few demonstrations, but its mechanisms are not well-understood. |
| Approach: | They characterize two ways in which LLMs leverage demonstrations to solve tasks with a few demonstrations. |
| Outcome: | The proposed model achieves non-trivial performance with only TR, and TR does not improve with larger models or more demonstrations. |
Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks (2026.acl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) enables models to adapt to new tasks via inference-time demonstrations. |
| Approach: | They propose a simple inference-stage enhancement method that reinforces task mapping transfer. |
| Outcome: | The proposed method strengthens task mapping transfer in multimodal models . it performs comparable to text-only ICL in zero-shot settings but degrades significantly under few-shot demonstrations. |
What Do Language Models Learn in Context? The Structured Task Hypothesis. (2024.acl-long)
Copied to clipboard
| Challenge: | Pre-trained large language models have exhibited an impressive ability to learn in context across various domains, e.g., code generation, education, medicine and even medicine. |
| Approach: | They taxonomize existing candidate theories into three competing hypotheses that explain LLMs’ ability to learn in context. |
| Outcome: | The proposed model can learn a task from in-context examples presented in a demonstration and generalize it to the prompt. |
MT2: Towards a Multi-Task Machine Translation Model with Translation-Specific In-Context Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Sentence-level translation, document-level and terminology constrained translations are important in machine translation. |
| Approach: | They propose a multi-task machine translation model that integrates translation memory sentences . they propose 'in-context learning' paradigm that allows translation-specific context learning . |
| Outcome: | The proposed model improves translation memory, document-level translation, and document-constrained translation tasks. |
Bag-of-Words Transfer: Non-Contextual Techniques for Multi-Task Learning (D19-61)
Copied to clipboard
| Challenge: | Existing approaches to multi-task learning take advantage of transfer among tasks . generative reconstruction of the observations is not included in the standard framework . |
| Approach: | They propose to use a syntactically-oblivious pooling encoder and pre-trained word embeddings to improve sentence-level representations. |
| Outcome: | The proposed techniques yield similar performance on a universe of task combinations while reducing training time and model size. |
Understanding In-Context Learning Beyond Transformers: An Investigation of State Space and Hybrid Architectures (2026.findings-acl)
Copied to clipboard
| Challenge: | In-context learning is an emergent ability from pretrained Large Language Models (LLMs). |
| Approach: | They perform in-depth evaluations of in-context learning on transformers and hybrid large language models using behavioral probing and intervention-based methods. |
| Outcome: | The proposed model performs well on state-of-the-art transformer, state-space, and hybrid large language models. |
An Empirical Study of In-context Learning in LLMs for Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies focus on optimizing translation quality, with limited attention to understanding specific aspects of ICL that influence the said quality. |
| Approach: | They conduct the first of its kind, exhaustive study of in-context learning for machine translation (MT) they establish that ICL is primarily example-driven and not instruction-driven . |
| Outcome: | The proposed model is based on examples and not instruction-driven learning. |
Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale (2023.acl-long)
Copied to clipboard
| Challenge: | 70% of attention heads and 20% of the feed forward networks can be removed with minimal decline in task performance. |
| Approach: | They propose to investigate whether in-context learning is not uniform across all components of a large language model. |
| Outcome: | The proposed model can remove 70% of attention heads and 20% of feed forward networks with minimal decline in task performance. |
A Learnable Skill Combination Strategy for Multi-task Learning in Natural Language Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | a novel multi-task learning framework for domain-specific natural language understanding tasks addresses these limitations by combing multiple tasks into a single framework. |
| Approach: | They propose a multi-task learning framework that decomposes the language model into modular skill components and employs a dynamic, learnable skill-combination mechanism to adaptively handle diverse tasks. |
| Outcome: | The proposed framework surpasses conventional multi-task learning approaches in performance. |