Take Off the Training Wheels! Progressive In-Context Learning for Effective Alignment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have explored the working mechanisms of In-Context Learning (ICL) however, they mainly focus on classification and simple generation tasks, limiting their broader application to more complex generation tasks in practice. |
| Approach: | They propose an efficient Progressive In-Context Alignment method that embeds the task function learned from demonstrations into the separator token representation. |
| Outcome: | The proposed method surpasses vanilla ICL and achieves comparable performance to other alignment tuning methods. |
Similar Papers
How Far Can In-Context Alignment Go? Exploring the State of In-Context Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have demonstrated that In-Context Learning (ICA) can align Large Language Models (LLMs) with human preferences without requiring parameter adjustments. |
| Approach: | They investigate the effectiveness of each part in enabling ICA to function effectively and examine how variants in these parts impact alignment performance. |
| Outcome: | The proposed model can comprehend human instructions without parameter adjustments. |
In-Context Learning Creates Task Vectors (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a powerful new learning paradigm for Large Language Models (LLMs). |
| Approach: | They propose to use a model with a prompt and a query to learn a mapping based on two examples to produce the output. |
| Outcome: | The proposed model can learn functions from a simple structure based on a training set and a single task vector calculated from the training set. |
FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Large pre-trained models are capable of few-shot in-context learning (ICL) however, concatenated demonstrations are often excessively long and require additional computation. |
| Approach: | They propose to apply fusion-in-decoder (FiD) models to perform few-shot in-context learning (ICL) they propose to use concatenation-based, early-fusion, intermediate- and late-fusion methods to improve efficiency . |
| Outcome: | The proposed methods outperform concatenation-based models on 11 held-out tasks. |
DeCoVec: Building Decoding Space based Task Vector for Large Language Models via In-Context Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to steering large language models require fine-tuning or manipulation of internal states, limiting their flexibility and scalability. |
| Approach: | They propose a framework that constructs task vectors directly in the decoding space by leveraging in-context learning. |
| Outcome: | The proposed framework outperforms standard few-shot baselines on TruthfulQA, Math-500, and AQUA-RAT with gains up to +5.50 accuracy. |
Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learning (2025.acl-long)
Copied to clipboard
Chengwei Qin, Wenhan Xia, Fangkai Jiao, Chen Chen, Yuchen Hu, Bosheng Ding, Ruirui Chen, Shafiq Joty
| Challenge: | Existing methods to train student models on the generated outputs of teacher models are not efficient for ICL. |
| Approach: | They propose to align the output of smaller (student) models with that of larger (teacher) models by incorporating a ranking loss and aligning the token-level output distribution. |
| Outcome: | The proposed model outperforms baseline models on a variety of tasks involving language understanding, reasoning, and coding. |
Beyond Demonstrations: Dynamic Vector Construction from Latent Representations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for In-Context Learning (ICL) are sensitive to ICL-specific factors and rely on heuristic-based injection positions. |
| Approach: | They propose a method that extracts task-relevant representations from large language models and reinjects them during inference. |
| Outcome: | The proposed method outperforms few-shot In-Context Learning (ICL) and LoRA methods without repeated demonstration processing. |
Pre-Training to Learn in Context (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models are not explicitly trained to learn in context. |
| Approach: | They propose a framework to enhance in-context learning by pre-training language models on a large collection of "intrinsic tasks" they evaluate the in-constitution learning performance of the model trained with PICL on seven widely-used text classification datasets and the Super-NaturalInstrctions benchmark . |
| Outcome: | The proposed framework outperforms larger language models with nearly 4x parameters on seven widely-used datasets and the Super-NaturalInstrctions benchmark. |
SADA: Bridging In-Context Learning and Fine-Tuning via State-Aligned Distillation Adapters (2026.acl-long)
Copied to clipboard
| Challenge: | Prompt-based in-context learning and parameter fine-tuning are dominant paradigms for incorporating external information into large language models, but they incur high inference costs or require expensive retraining. |
| Approach: | They propose to convert prompts into temporary adapter weights to bridge this gap by converting prompts to temporary adapters. |
| Outcome: | The proposed model outperforms baselines on long-context language modeling and downstream NLU and summarization benchmarks while significantly reducing memory footprint and latency. |
Enhancing In-Context Learning via Implicit Demonstration Augmentation (2024.acl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a new paradigm for pre-trained language models that can make predictions for unseen inputs without updating parameters. |
| Approach: | They propose a method that enables a model to augmented copies of a demonstration by leveraging their deep feature distribution and a logit calibration mechanism. |
| Outcome: | The proposed method significantly improves the average and worst-case accuracy across diverse PLMs and tasks. |
What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can perform in-context learning (ICL) with only a few demonstrations, but its mechanisms are not well-understood. |
| Approach: | They characterize two ways in which LLMs leverage demonstrations to solve tasks with a few demonstrations. |
| Outcome: | The proposed model achieves non-trivial performance with only TR, and TR does not improve with larger models or more demonstrations. |