OptiSeq: Ordering Examples On-The-Fly for In-Context Learning (2025.findings-emnlp)
Copied to clipboard
Rahul Atul Bhope, Praveen Venkateswaran, K. R. Jayaram, Vatche Isahagian, Vinod Muthusamy, Nalini Venkatasubramanian
| Challenge: | In-context-learning (ICL) is fragile and requires a lot of examples to perform. |
| Approach: | They propose a purely inference-time, dataset-free optimization method that efficiently determines the best example order. |
| Outcome: | The proposed method improves in-context-learning accuracy by 5.5 - 10.5 percentage points across multiple tasks. |
Similar Papers
What Makes a Good Order of Examples in In-Context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) demonstrate impressive few-shot learning capabilities via in-context learning (ICL). |
| Approach: | They propose to use unlabeled data to evaluate order performance . they propose to filter out subsets of orders with label fairness and select the most influential order for each test instance. |
| Outcome: | The proposed method is superior over strong baselines and validates generalizability across settings. |
In-Context Example Ordering Guided by Label Distributions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Prior work has shown that ICL is sensitive to different natural language instructions and different orderings of in-context examples. |
| Approach: | They propose two principles for in-context example ordering guided by model’s probability predictions. |
| Outcome: | The proposed model outperforms baseline models on 13 text classification datasets and nine autoregressive LLMs with 700M to 13B parameters. |
NICE: To Optimize In-Context Examples or Not? (2024.acl-long)
Copied to clipboard
| Challenge: | Recent work shows that in-context learning and optimization of in-const examples (ICE) can improve the accuracy of large language models on a wide range of tasks. |
| Approach: | They propose a task-specific metric called Normalized Invariability to Choice of Examples (NICE) metric measures the learnability of tasks from a given instruction and provides a heuristic to decide whether to optimize ICE for a new task. |
| Outcome: | The proposed metric predicts the utility of optimizing ICE for a given task compared to random ICE. |
Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a meta-optimization process that affects performance . we develop a batch-based inference algorithm that is order-agnostic to ICL examples . |
| Approach: | They develop an order-agnostic inference algorithm that aggregates ICL examples in batches . they find it outperforms most permutations of ICL, and it even exceeds the best order . |
| Outcome: | The proposed method outperforms standard ICL examples while reducing computational resources. |
Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering (2023.acl-long)
Copied to clipboard
| Challenge: | In-context learning is a common practice to randomly sample examples to serve as context. |
| Approach: | They propose a new principle for in-context learning that helps each sample find an in-constitut example organization that can derive the correct prediction. |
| Outcome: | The proposed method achieves 40% relative improvement over the common practice setting. |
How Far Can In-Context Alignment Go? Exploring the State of In-Context Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have demonstrated that In-Context Learning (ICA) can align Large Language Models (LLMs) with human preferences without requiring parameter adjustments. |
| Approach: | They investigate the effectiveness of each part in enabling ICA to function effectively and examine how variants in these parts impact alignment performance. |
| Outcome: | The proposed model can comprehend human instructions without parameter adjustments. |
More Samples or More Prompts? Exploring Effective Few-Shot In-Context Learning for LLMs with In-Context Sampling (2024.findings-naacl)
Copied to clipboard
Bingsheng Yao, Guiming Chen, Ruishi Zou, Yuxuan Lu, Jiachen Li, Shao Zhang, Yisi Sang, Sijia Liu, James Hendler, Dakuo Wang
| Challenge: | Existing studies on LLM prompting focus on selecting a better set of data samples inside one single prompt input, but why not design and leverage multiple ICL prompts together to further improve the LLM’s performance? |
| Approach: | They propose a low-resource LLM prompting technique to optimize the construction of multiple ICL prompt inputs to produce confident predictions. |
| Outcome: | The proposed technique can produce confident predictions by optimizing the construction of multiple ICL prompt inputs on four NLI datasets and one QA dataset. |
Data Curation Alone Can Stabilize In-context Learning (2023.acl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a new paradigm for few-shot learning with pretrainable large language models . however, randomly sampling examples from a training set leads to high variance in performance . |
| Approach: | They propose two methods to select training examples from a training set and then carefully curate them from corresponding subsets. |
| Outcome: | The proposed method improves accuracy over sampling from the entire training set. |
Addressing Order Sensitivity of In-Context Demonstration Examples in Causal Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | In-context learning is a popular paradigm in natural language processing, but its performance can be significantly influenced by the order of in-concept demonstration examples. |
| Approach: | They propose an unsupervised fine-tuning method to reduce the sensitivity of causal language models to the order of in-context demonstration examples. |
| Outcome: | The proposed method reduces the sensitivity of CausalLMs to the order of in-context examples and exhibits robust generalizability. |
Strategic Demonstration Selection for Improved Fairness in LLM In-Context Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies highlight the effectiveness of using in-context learning (ICL) to steer large language models in processing tabular data. |
| Approach: | They propose a method that uses clustering and evolutionary strategies to curate a representative sample set from training data. |
| Outcome: | The proposed method significantly improves fairness across various metrics, showing its efficacy in real-world scenarios. |