Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized inference across diverse natural language tasks, with larger models performing better but at higher computational costs. |
| Approach: | They propose a confidence-driven strategy that dynamically selects the most suitable model based on confidence estimates. |
| Outcome: | The proposed approach reduces token usage by approximately 60% and improves cost efficiency on the Massive Multitask Language Understanding (MMLU) benchmark. |
Similar Papers
A Survey of Confidence Estimation and Calibration in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks in various domains, but they can be unreliable due to factual errors in their generations. |
| Approach: | They summarize recent advances in LLM confidence estimation and calibration and outline their main lessons learned. |
| Outcome: | The proposed methods can be used to assess the reliability of models and to calibrate them across tasks. |
All Roads Lead to Rome: Graph-Based Confidence Estimation for Large Language Model Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for confidence estimation are primarily designed for factual QA tasks and fail to generalize to reasoning tasks. |
| Approach: | They propose a set of training-free, graph-based confidence estimation methods tailored to reasoning tasks that exploit graph properties such as centrality, path convergence, and path weighting. |
| Outcome: | The proposed methods improve confidence estimation and performance on two downstream tasks. |
Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems (2025.findings-emnlp)
Copied to clipboard
Aakriti Agrawal, Rohith Aralikatti, Anirudh Satheesh, Souradip Chakraborty, Amrit Singh Bedi, Furong Huang
| Challenge: | Existing approaches to selecting reliable responses from multiple LLMs often depend on external verifiers, human evaluators, or self-consistency techniques. |
| Approach: | They propose a calibrated log-likelihood-based selection framework to improve multi-LLM performance. |
| Outcome: | The proposed method outperforms majority voting and exceeds self-consistency performance when using a large number of model calls. |
MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models (2025.findings-acl)
Copied to clipboard
Boyang Xue, Hongru Wang, Rui Wang, Sheng Wang, Zezhong Wang, Yiming Du, Bin Liang, Wenxuan Zhang, Kam-Fai Wong
| Challenge: | Existing studies on LLM confidence estimations in languages other than English have been limited to English. |
| Approach: | They propose to use question-related language to prompt LLMs to assess their confidence in large language models. |
| Outcome: | The proposed model improves on question-related language prompts for LS tasks, while English exhibits notable linguistic dominance in confidence estimations. |
Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used in high-stakes areas such as healthcare, law, and education. |
| Approach: | They propose a concept of Confidence-Probability Alignment that connects an LLM’s internal confidence to the confidence conveyed in the model’s response when explicitly asked about its certainty. |
| Outcome: | The proposed model shows the strongest confidence-probability alignment across a wide range of tasks. |
Confidence-Calibrated Small-Large Language Model Collaboration for Cost-Efficient Reasoning (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have superior reasoning capabilities compared to small language models, but incur substantially higher inference costs. |
| Approach: | They propose a system that cascades an LLM with an SLM to achieve a balance between accuracy and cost in complex reasoning tasks. |
| Outcome: | The proposed system improves the SLM’s reasoning ability and confidence calibration across diverse datasets and model backbones. |
Evaluating Large Language Models for Confidence-based Check Set Selection (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models have shown promise in automating high-labor data tasks, but their tendency to answer despite uncertainty and their difficulty handling long input contexts robustly are key challenges for adoption. |
| Approach: | They propose to use LLMs to prioritize information needing human judgment to identify low-confidence outputs for human review through "check set selection" using social media monitoring, they define the "check sets" as a list of tweets escalated to the disaster manager when the LLM has the least confidence. |
| Outcome: | The proposed approach outperforms random-sample check set selection in disaster tweet classification. |
SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models struggle with complex tasks such as factually-grounded reasoning and planning due to inherent training biases, model size constraints, and the quality or diversity of pre-training datasets. |
| Approach: | They propose a novel algorithm to select the most suitable LLMs from a large pool and use it to efficiently generalize and perform tasks. |
| Outcome: | The proposed model outperforms existing ensemble-based baselines and achieves competitive performance with similarly sized top-performing LLMs while maintaining efficiency. |
AdaptEvolve: Improving Efficiency of Evolutionary AI Agents through Adaptive Model Selection (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing routing strategies rely on static heuristics or external controllers to optimize performance. |
| Approach: | They propose a framework that leverages intrinsic generation confidence to estimate solvability. |
| Outcome: | Empirical results show that confidence-driven selection yields favorable Pareto frontier . computational cost of state-of-the-art large language models remains a key barrier to scalable deployment . |
Can Unconfident LLM Annotations Be Used for Confident Conclusions? (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown high agreement with human raters across a variety of tasks, demonstrating potential to ease the challenges of human data collection. |
| Approach: | They propose a method that combines LLM annotations and LLM confidence indicators to strategically select which human annotations to use. |
| Outcome: | The proposed method produces accurate estimates and valid confidence intervals while reducing the number of human annotations by over 25%. |