COM-BOM: Bayesian Exemplar Search for Efficiently Exploring the Accuracy-Calibration Pareto Frontier (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior exemplar selection methods focus on maximizing predictive accuracy, neglecting model calibration. |
| Approach: | They propose to use a Bayesian optimization algorithm to optimize for predictive accuracy and calibration. |
| Outcome: | The proposed algorithm beats or matches baselines on multiple tasks from un-saturated MMLU-pro benchmarks while requiring minimal number of API calls. |
Similar Papers
EXPLORA: Efficient Exemplar Subset Selection for Complex Reasoning (2024.emnlp-main)
Copied to clipboard
Kiran Purohit, Venktesh V, Raghuram Devalla, Krishna Yerragorla, Sourangshu Bhattacharya, Avishek Anand
| Challenge: | Recent advances in large language models (LLMs) have enabled in-context learning (ICL) a critical challenge in ICL is the selection of optimal exemplars . |
| Approach: | They propose an algorithm for static exemplar subset selection for reasoning tasks . they propose a method that estimates parameters without incorporating confidence information . |
| Outcome: | The proposed method significantly reduces the number of LLM calls to 11% of those required by state-of-the-art methods and achieves a substantial performance improvement of 12.24%. |
Precise Model Benchmarking with Only a Few Observations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Accurate evaluation of large language models is crucial for identifying their strengths and weaknesses. |
| Approach: | They propose an empirical Bayes estimator that balances direct and regression estimates for each subgroup separately, improving the precision of subgroup-level estimates of model performance. |
| Outcome: | The proposed model reduces the mean squared error by up to 50% on multiple datasets. |
Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to optimize large language models with human preferences suffer from preference conflicts in the data. |
| Approach: | They propose to construct Pareto-optimal responses to resolve preference conflicts by using a self-improving DPO framework that enables LLMs to self-generate and select Paret-optimized responses. |
| Outcome: | The proposed framework achieves superior Pareto Front performance over baselines on two datasets. |
An End-to-End Submodular Framework for Data-Efficient In-Context Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in natural language tasks leverage the emergent In-Context Learning ability of pretrained Large Language Models (LLMs). |
| Approach: | They propose a framework for exemplar selection for in-context learning that uses a pool-based active learning approach to select Diverse and informative exemplars from the target tasks’ unlabeled pool. |
| Outcome: | The proposed framework outperforms existing methods for data annotation and similarity-based methods for test query-specific exemplar retrieval on 7 different NLP datasets and 5 LLMs of varying complexities. |
STARE at the Structure: Steering ICL Exemplar Selection with Structural Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for incontext learning often overlook structural alignment, leading to poor generalization and suboptimal performance. |
| Approach: | They propose a two-stage exemplar selection strategy that achieves a strong balance between efficiency, generalizability and performance. |
| Outcome: | The proposed method outperforms baselines on semantic parsing tasks on four benchmarks. |
Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging (2026.findings-acl)
Copied to clipboard
| Challenge: | We show that the "alignment tax" of post-training is framed as a drop in task accuracy. |
| Approach: | They propose a more holistic view of the alignment tax by framing it as a drop in accuracy and a degradation of model calibration. |
| Outcome: | The proposed method improves accuracy beyond both parents while recovering calibration lost during alignment. |
Bayesian Learning for Neural Dependency Parsing (N19-1)
Copied to clipboard
| Challenge: | Several approaches for dependency parsing in the small data regime have been proposed. |
| Approach: | They propose to use stochastic gradient Langevin dynamics to generate samples from the approximated posterior to overcome the computational and statistical costs of the approximate inference step. |
| Outcome: | The proposed model outperforms the biaffine model on 6 languages with less than 5k training instances and improves across five languages. |
SAJA: A Simple Approach to Judge Alignment for LLM-as-a-Judge (2026.acl-industry)
Copied to clipboard
| Challenge: | Current approaches to evaluate text at scale require multiple calls and per-dataset prompt tuning. |
| Approach: | They propose a model-agnostic approach to evaluate judge alignment that uses a lightweight calibration head. |
| Outcome: | a new model with SAJA matches more complex systems across four evaluation paradigms . it outperforms uncalibrated models on MT-Bench pairwise preference and competitive performance on five classification benchmarks compared to uncalibred models . |
Efficient and Accurate Prompt Optimization: the Benefit of Memory in Exemplar-Guided Reflection (2025.acl-long)
Copied to clipboard
Cilin Yan, Jingyun Wang, Lin Zhang, Ruihui Zhao, Xiaopu Wu, Kai Xiong, Qingsong Liu, Guoliang Kang, Yangyang Kang
| Challenge: | Recent work utilizes feedbacks generated from erroneous cases to guide prompt optimization . previous methods rely on computational resources and powerful GPUs . |
| Approach: | They propose an automatic prompt engineering method that leverages feedbacks from erroneous cases to guide prompt optimization. |
| Outcome: | The proposed method surpasses state-of-the-art methods with less steps and lower computational resources. |
Writing Like the Best: Exemplar-Based Expository Text Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for expository text generation rely on extensive domain corpora and are difficult to adapt topic-specific content. |
| Approach: | They propose a task to generate an expository text on a new topic using an exemplar on . they propose Adaptive Imitation and a Recurrent Plan-then-Adapt framework that leverages large language models for effective adaptive imitation. |
| Outcome: | The proposed framework surpasses existing baselines in producing factual, consistent, and relevant texts for this task. |