Estimating Large Language Model Capabilities without Labeled Test Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have shown impressive ability to perform in-context learning from only a few examples, but their accuracy varies widely from task to task. |
| Approach: | They propose a method that trains a meta-model using LLM confidence scores as features to perform ICL accuracy estimation. |
| Outcome: | The proposed method improves over baselines across 7 out of 12 settings and achieves the same accuracy as evaluating on 40 sampled examples per task. |
Similar Papers
ICLEval: Evaluating In-Context Learning Ability of Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing evaluation frameworks focus on language abilities and knowledge, often overlooking the assessment of ICL ability. |
| Approach: | They propose to evaluate the ICL ability of Large Language Models (LLMs) using the ICLEval benchmark. |
| Outcome: | The proposed benchmark demonstrates that ICL ability is universally present in different LLMs and model size is not the sole determinant of ICL efficacy. |
An Empirical Study of In-context Learning in LLMs for Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent studies focus on optimizing translation quality, with limited attention to understanding specific aspects of ICL that influence the said quality. |
| Approach: | They conduct the first of its kind, exhaustive study of in-context learning for machine translation (MT) they establish that ICL is primarily example-driven and not instruction-driven . |
| Outcome: | The proposed model is based on examples and not instruction-driven learning. |
Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanism (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models exhibit remarkable in-context learning (ICL) capabilities, but the underlying working mechanism of ICL remains unclear. |
| Approach: | They propose a Two-Dimensional Coordinate System that unifies both views into a systematic framework that explains the behavior of ICL through two orthogonal variables: whether similar examples are presented in the demonstrations and whether LLMs can recognize the task. |
| Outcome: | The proposed method can interpret ICL for generation tasks effectively. |
Tutor-ICL: Guiding Large Language Models for Improved In-Context Learning Performance (2024.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a dominant paradigm in natural language processing. |
| Approach: | They propose a prompting method for classification tasks using exemplar answers in a *comparative format' they also propose introducing a test instance before the exemplars to improve performance . |
| Outcome: | The proposed method achieves up to 13.76% increase in accuracy on classification tasks across decoder-only and encoder-decoder LLMs. |
From Cross-Task Examples to In-Task Prompts: A Graph-Based Pseudo-Labeling Framework for In-context Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning (ICL) enables large language models to perform novel tasks without parameter updates by conditioning on a few input-output examples. |
| Approach: | They propose a cost-efficient two-stage pipeline that reduces reliance on LLMs for data labeling. |
| Outcome: | The proposed pipeline reduces reliance on LLMs for data labeling . it leverages readily available cross-task examples to prompt an LLM and pseudo-label a small set of target task instances. |
IAD: In-Context Learning Ability Decoupler of Large Language Models in Meta-Training (2024.lrec-main)
Copied to clipboard
| Challenge: | In-context Learning (ICL) is a paradigm in which LLMs acquire task-specific knowledge by processing input-output pairs provided as prompts. |
| Approach: | They propose an In-context learning Ability Decoupler to separate ICL ability from general ability of LLMs in meta-training phase . they first identify parameters suitable for ICL by transference-driven gradient importance and propose a new max-margin loss to emphasize the separation of the two abilities. |
| Outcome: | The proposed model separates the ICL ability from the general ability of LLMs in the meta-training phase, where the I-related parameters are tuned to adapt for ICL tasks. |
Large Language Models are Miscalibrated In-Context Learners (2025.findings-acl)
Copied to clipboard
| Challenge: | In-context Learning and Supervised Fine-Tuning have emerged as pre-dominant methodologies for machine learning and NLP. |
| Approach: | They propose to use self-ensembling to improve both performance and calibration of language models. |
| Outcome: | The proposed learning paradigms can achieve better calibration and better performance than the previous learning paradigm. |
What Do Language Models Learn in Context? The Structured Task Hypothesis. (2024.acl-long)
Copied to clipboard
| Challenge: | Pre-trained large language models have exhibited an impressive ability to learn in context across various domains, e.g., code generation, education, medicine and even medicine. |
| Approach: | They taxonomize existing candidate theories into three competing hypotheses that explain LLMs’ ability to learn in context. |
| Outcome: | The proposed model can learn a task from in-context examples presented in a demonstration and generalize it to the prompt. |
Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning Perspective (2026.acl-long)
Copied to clipboard
Bishwamittra Ghosh, Soumi Das, Till Speicher, Qinyuan Wu, Mohammad Aflah Khan, Deepak Garg, Krishna P. Gummadi, Evimaria Terzi
| Challenge: | Prior studies comparing FT and ICL have yielded mixed and inconclusive results due to inconsistent experimental setups. |
| Approach: | They propose a formal language learning task with precise language boundaries, controlled string sampling, and no data contamination to enable a rigorous comparison. |
| Outcome: | The proposed task offers precise language boundaries, controlled string sampling, and no data contamination. |
Uncertainty Quantification for In-Context Learning of Large Language Models (2024.naacl-long)
Copied to clipboard
Chen Ling, Xujiang Zhao, Xuchao Zhang, Wei Cheng, Yanchi Liu, Yiyou Sun, Mika Oishi, Takao Osaki, Katsushi Matsuda, Jie Ji, Guangji Bai, Liang Zhao, Haifeng Chen
| Challenge: | Existing studies on in-context learning have focused on quantifying the uncertainty associated with the model's response, but they neglect the complexity of the LLM and the uniqueness of in-constitut learning. |
| Approach: | They propose a method to quantify the uncertainty associated with in-context learning and propose corresponding estimation method to quantify both types of uncertainties. |
| Outcome: | The proposed method offers an unsupervised way to understand the prediction of in-context learning in a plug-and-play fashion. |