Crossroads of Optimization under Uncertainty: How to Choose the Optimal Model (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to Optimization under Uncertainty (OuU) have inherent limitations and advantages. |
| Approach: | They propose a framework that automates the modeling and solving of six types of uncertainty models and generates mapping pairs to explore the potential relationship between optimization problems and optimal models. |
| Outcome: | The proposed framework achieves superior performance even on specific model types, with correlation analysis showing that data scale and specific scenario significantly influence model selection. |
Similar Papers
Rethinking LLM Uncertainty: A Multi-Agent Approach to Estimating Black-Box Model Uncertainty (2025.findings-emnlp)
Copied to clipboard
Yu Feng, Phu Mon Htut, Zheng Qi, Wei Xiao, Manuel Mager, Nikolaos Pappas, Kishaloy Halder, Yang Li, Yassine Benajiba, Dan Roth
| Challenge: | Existing methods to gauge model’s uncertainty through self-consistency in responses to the target query are misleading: an LLM may confidently provide an incorrect answer to a target query, yet give a confident and accurate answer to that same query when answering a knowledge-preserving perturbation of the query. |
| Approach: | They propose a method that uses multi-agent interaction to estimate black-box LLMs' uncertainty. |
| Outcome: | The proposed method outperforms existing self-consistency based methods and improves hallucination detection. |
Do not Abstain! Identify and Solve the Uncertainty (2025.acl-long)
Copied to clipboard
| Challenge: | Existing solutions rely on evasive responses when confronting uncertain scenarios. |
| Approach: | They propose a benchmark to assess LLMs' ability to recognize and address uncertainty . they generate context-aware inquiries that highlight the confusing aspect of the original query . |
| Outcome: | Experiments with ConfuseBench show that LLMs struggle to identify root cause of uncertainty and solve it. |
Reconsidering LLM Uncertainty Estimation Methods in the Wild (2025.acl-long)
Copied to clipboard
Yavuz Faruk Bakman, Duygu Nur Yaldiz, Sungmin Kang, Tuo Zhang, Baturalp Buyukates, Salman Avestimehr, Sai Praneeth Karimireddy
| Challenge: | Existing studies evaluate UE methods in short-form QA settings, but real-world deployment presents several challenges. |
| Approach: | They examine UE methods' sensitivity to decision threshold selection and their robustness to query transformations such as typos and adversarial prompts. |
| Outcome: | The proposed methods exhibit robustness against typos, adversarial prompts, and prior chat history, and are highly susceptible to adversarials. |
Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities (2026.acl-long)
Copied to clipboard
Changdae Oh, Seongheon Park, To Eun Kim, Jiatong Li, Wendi Li, Samuel Yeh, Sean Du, Hamed Hassani, Paul Bogdan, Dawn Song, Sharon Li
| Challenge: | Uncertainty quantification (UQ) for large language models is a key building block for daily applications. |
| Approach: | They propose a general formulation of agent UQ that subsumes broad classes of existing UQ setups. |
| Outcome: | The proposed framework is based on the first general formulation of agent UQ that subsumes broad classes of existing setups. |
Beyond "I Don’t Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty (2026.acl-long)
Copied to clipboard
| Challenge: | Prior studies treat refusal as a generic "I don't know" lack of distinction limits downstream action decisions like requesting clarification or invoking external tools. |
| Approach: | They propose a benchmark to evaluate explicit uncertainty attribution in large language models. |
| Outcome: | The proposed method improves uncertainty attribution while preserving answer accuracy. |
Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have emerged as powerful tools for a wide range of tasks, from * Equal Contribution. |
| Approach: | They propose a framework that enhances communication efficiency and task effectiveness in LLM-based multi-agent systems through training. |
| Outcome: | The proposed framework improves communication efficiency and task effectiveness on multi-agent tasks with 2.8x performance gain with less than 10% tokens on tasks requiring heavy information exchange. |
A Survey of Uncertainty Estimation Methods on Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities but could produce biased, hallucinated, or non-factual responses. |
| Approach: | They propose to conduct extensive experimental evaluations of LLM uncertainty estimation methods . large language models have demonstrated remarkable capabilities across tasks . |
| Outcome: | The proposed method could produce biased, hallucinated, or non-factual responses . a lack of comprehensive surveys on LLM uncertainty estimation is a problem . |
Benchmarking Uncertainty Metrics for LLM Target-Aware Search (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing uncertainty metrics for LLM search methods do not capture the diverse types of uncertainty needed to guide different optimization goals. |
| Approach: | They propose a framework for uncertainty benchmarking that captures four different uncertainty types . the uncertainty types Answer, Correctness, Aleatoric, and Epistemic serve different optimization goals . |
| Outcome: | The proposed framework identifies four different uncertainty types . the uncertainty types serve different optimization goals in LLM search . |
Adaptive Prompt Optimization for Open-Ended Tasks: Uncertainty Preference as a Secondary Signal (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent training-free prompt optimizers treat performance as maximizing a single scalar score and ignore a second signal that the desired style is task dependent. |
| Approach: | They propose a semantic-entropy-based method that uses task uncertainty to guide prompt optimization by selecting high-entropicy candidates for creative tasks and low-energetic candidates for conservative ones. |
| Outcome: | The proposed method outperforms baselines on MT-Bench subsets and integrates easily into existing prompt optimizers. |
Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems (2025.findings-emnlp)
Copied to clipboard
Aakriti Agrawal, Rohith Aralikatti, Anirudh Satheesh, Souradip Chakraborty, Amrit Singh Bedi, Furong Huang
| Challenge: | Existing approaches to selecting reliable responses from multiple LLMs often depend on external verifiers, human evaluators, or self-consistency techniques. |
| Approach: | They propose a calibrated log-likelihood-based selection framework to improve multi-LLM performance. |
| Outcome: | The proposed method outperforms majority voting and exceeds self-consistency performance when using a large number of model calls. |