Soft Head Selection for Injecting ICL-Derived Task Embeddings (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are commonly adapted to downstream tasks using parameter-efficient fine-tuning (PEFT) or in-context learning (ICL). |
| Approach: | They propose a gradient-based method that derives task-specific embeddings from activations using few-shot prompts and injects them during inference. |
| Outcome: | The proposed method outperforms existing methods on open-ended generation, reasoning, and natural language understanding tasks while using fewer trainable parameters. |
Similar Papers
Selecting Demonstrations for Many-Shot In-Context Learning via Gradient Matching (2025.findings-acl)
Copied to clipboard
| Challenge: | In-Context Learning (ICL) empowers Large Language Models for rapid task adaptation without fine-tuning. |
| Approach: | They propose a method that aligns fine-tuning gradients between entire training set and selected examples to enable in-context learning and fine-uning. |
| Outcome: | The proposed method outperforms random selection on large LLMs from 4-shot to 128-shot scenarios across 9 datasets. |
Learning Task Representations from In-Context Learning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for generalizing tasks to modalities beyond text fail to generalize effectively to linguistic tasks. |
| Approach: | They propose a method for encoding task information in ICL prompts as a function of attention heads within the transformer architecture. |
| Outcome: | The proposed method extracts task-specific information from in-context demonstrations and excels in both text and regression tasks. |
PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from related Example Banks (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive few-shot learning capabilities through in-context learning. |
| Approach: | They propose a novel Alternating Minimization approach for example selection that improves ICL performance on low-resource Indic languages. |
| Outcome: | The proposed approach outperforms existing frameworks for retrieving examples on low-resource Indic languages. |
Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Recent learning-based demonstration selection methods have proven beneficial to in-context learning (ICL) by choosing more useful exemplars. |
| Approach: | They propose two methods to capture task-agnostic similarities between input and output of LLMs. |
| Outcome: | The proposed methods integrate task-agnostic similarities of different levels between input and output of exemplars and test cases to eliminate costly data collection. |
Hit the Nail on the Head: Parameter-Efficient Multi-task Tuning via Human Language Intervention (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that PEFT on small pre-trained language models improves multitasking capabilities. |
| Approach: | They propose a multi-task learning framework that enables transfer of prior knowledge across tasks . they attach task descriptions to input samples and map them to task embeddings . |
| Outcome: | The proposed method improves performance on a T5 model and in decoder-only models . |
One Task Vector is not Enough: A Large-Scale Study for In-Context Learning (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing studies limit comprehensive analysis of large language models based on task vectors . recent work points to "task vectors" as mechanism for encoding task rules . |
| Approach: | They propose a novel task vector with 30 input-output pairs for in-context learning . they use a few prompt-based examples to adapt to new tasks without weight updates . |
| Outcome: | Experiments with Llama-3-8B on QAF show task vector performance peaks at intermediate layer . complex tasks rely on multiple, subtask-specific vectors rather than a single vector . |
Beyond Demonstrations: Dynamic Vector Construction from Latent Representations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for In-Context Learning (ICL) are sensitive to ICL-specific factors and rely on heuristic-based injection positions. |
| Approach: | They propose a method that extracts task-relevant representations from large language models and reinjects them during inference. |
| Outcome: | The proposed method outperforms few-shot In-Context Learning (ICL) and LoRA methods without repeated demonstration processing. |
In-Context Learning with Iterative Demonstration Selection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing literature has highlighted the importance of selecting examples that are diverse or semantically similar to the test sample . Existing studies have shown that the optimal selection dimension, i.e., diversity or similarity, is task-specific. |
| Approach: | They propose to use zero-shot chain-of-thought reasoning to iteratively select examples that are diverse but still strongly correlated with the test sample as ICL demonstrations. |
| Outcome: | The proposed method outperforms existing demonstration selection methods on reasoning, question answering, and topic classification tasks. |
FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Large pre-trained models are capable of few-shot in-context learning (ICL) however, concatenated demonstrations are often excessively long and require additional computation. |
| Approach: | They propose to apply fusion-in-decoder (FiD) models to perform few-shot in-context learning (ICL) they propose to use concatenation-based, early-fusion, intermediate- and late-fusion methods to improve efficiency . |
| Outcome: | The proposed methods outperform concatenation-based models on 11 held-out tasks. |
STARE at the Structure: Steering ICL Exemplar Selection with Structural Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for incontext learning often overlook structural alignment, leading to poor generalization and suboptimal performance. |
| Approach: | They propose a two-stage exemplar selection strategy that achieves a strong balance between efficiency, generalizability and performance. |
| Outcome: | The proposed method outperforms baselines on semantic parsing tasks on four benchmarks. |