| Challenge: | In-context learning is a new paradigm in natural language understanding . large pre-trained language models can be expensive to update . |
| Approach: | They propose an efficient method for retrieving training examples as prompts from annotated data and an LM. |
| Outcome: | The proposed method outperforms prior work and multiple baselines on three sequence-to-sequence tasks. |
Similar Papers
Learning to Retrieve In-Context Examples for Large Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to improve in-context learning performance are highly sensitive to the quality of the incontext examples provided. |
| Approach: | They propose a framework to iteratively train dense retrievers that can identify high-quality in-context examples for large language models. |
| Outcome: | The proposed model improves performance by retrieving examples with similar patterns, and the gains are consistent across LLMs of varying sizes. |
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts (2021.naacl-main)
Copied to clipboard
| Challenge: | Pretrained language models retain factual knowledge that can be extracted with a sentential prompt. |
| Approach: | They propose to learn prompts by gradient descent, either fine-tuning prompts or starting from random initialization. |
| Outcome: | The proposed approach outperforms existing methods on English LMs and tasks. |
Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models demonstrate remarkable ability for learning to solve new tasks from a few examples. |
| Approach: | They propose to use templates to aggregate model predictions across multiple templates to improve model performance. |
| Outcome: | The proposed model ensembles boost model predictions while being robust to the choice of random set of templates. |
How Can We Know What Language Models Know? (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent work examines knowledge contained in language models by having the LM fill in the blanks of prompts such as “Obama is a __ by profession”. |
| Approach: | They propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. |
| Outcome: | The proposed methods improve accuracy from 31.1% to 39.6% on the LAMA benchmark for extracting relational knowledge from LMs. |
Prompt Optimization via Adversarial In-Context Learning (2024.acl-long)
Copied to clipboard
Do Long, Yiran Zhao, Hannah Brown, Yuxi Xie, James Zhao, Nancy Chen, Kenji Kawaguchi, Michael Shieh, Junxian He
| Challenge: | Existing methods to optimize prompts for in-context learning are based on adversarial learning and are computationally efficient and extensible to other LLMs and tasks. |
| Approach: | They propose a method to optimize prompts for in-context learning by a generator and a discriminator. |
| Outcome: | The proposed method improves state-of-the-art prompt optimization techniques on 13 generation and classification tasks including summarization, arithmetic reasoning, machine translation, data-to-text generation, and the MMLU and big-bench hard benchmarks. |
Exploiting Language Model Prompts Using Similarity Measures: A Case Study on the Word-in-Context Task (2022.acl-short)
Copied to clipboard
| Challenge: | Existing few-shot approaches fail on the semantic distinction task of the Word-in-Context dataset. |
| Approach: | They propose a prompt-based approach which boosts few-shot performance to the level of fully supervised methods by using similarity metrics. |
| Outcome: | The proposed technique boosts few-shot performance to the level of fully supervised methods. |
More Samples or More Prompts? Exploring Effective Few-Shot In-Context Learning for LLMs with In-Context Sampling (2024.findings-naacl)
Copied to clipboard
Bingsheng Yao, Guiming Chen, Ruishi Zou, Yuxuan Lu, Jiachen Li, Shao Zhang, Yisi Sang, Sijia Liu, James Hendler, Dakuo Wang
| Challenge: | Existing studies on LLM prompting focus on selecting a better set of data samples inside one single prompt input, but why not design and leverage multiple ICL prompts together to further improve the LLM’s performance? |
| Approach: | They propose a low-resource LLM prompting technique to optimize the construction of multiple ICL prompt inputs to produce confident predictions. |
| Outcome: | The proposed technique can produce confident predictions by optimizing the construction of multiple ICL prompt inputs on four NLI datasets and one QA dataset. |
How Does In-Context Learning Help Prompt Tuning? (2024.findings-eacl)
Copied to clipboard
| Challenge: | a growing number of parameter-efficient adaptation methods are needed to fine-tune large language models. |
| Approach: | They propose a method that combines prompt tuning and in-context learning to improve prompt tuning by concatenating a natural language demonstration with learned prompt embeddings. |
| Outcome: | The proposed method outperforms prompt tuning and prompt tuning on five language generation tasks. |
Self-Instructed Derived Prompt Generation Meets In-Context Learning: Unlocking New Potential of Black-Box LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Existing prompt refinement methods suffer from semantic inconsistencies and fail to maintain users’ real intent. |
| Approach: | They propose a self-instructed in-context learning framework that generates reliable derived prompts while keeping semantic consistency with original prompts. |
| Outcome: | The proposed framework generates better derived prompts and significantly enhances LLMs’ ability to deliver more effective responses. |
Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on in-context learning mechanisms are not consistent . current research identifies two main approaches to explain the ICL mechanism . |
| Approach: | They propose a framework for evaluating in-context learning mechanisms by focusing on regression tasks. |
| Outcome: | The proposed framework can solve regression problems and then measure the extent to which the LLM retrieves its internal knowledge versus learning from in-context examples. |