Evaluating Uncertainty Quantification Methods in Argumentative Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | ArgLLMs are an explainable LLM framework for decision-making based on computational argumentation in which uncertainty quantification plays a critical role. |
| Approach: | They propose to integrate LLM UQ methods into argumentative LLMs to evaluate their performance on claim verification tasks. |
| Outcome: | The proposed method outperforms more complex approaches on claim verification tasks. |
Similar Papers
SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Large language models have a tendency to make confidently wrong predictions, highlighting the need for uncertainty quantification (UQ) . previous studies focused on aleatoric uncertainty, but the full spectrum of uncertainties, including epistemic, remains inadequately explored. |
| Approach: | They propose a method to quantify uncertainty in large language models (LLMs) they use a set of perturbations and an aggregation module to generalize the method. |
| Outcome: | The proposed method improves model uncertainty calibration and reduces expected calibration error by 50% on average. |
Uncertainty Quantification for Large Language Models (2025.acl-tutorials)
Copied to clipboard
| Challenge: | Large language models (LLMs) produce hallucinations, which undermine user trust and reliability. |
| Approach: | This tutorial offers the first systematic introduction to uncertainty quantification (UQ) for LLMs in text generation tasks. |
| Outcome: | The proposed framework provides tools for communicating the reliability of a model answer. |
Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities (2026.acl-long)
Copied to clipboard
Changdae Oh, Seongheon Park, To Eun Kim, Jiatong Li, Wendi Li, Samuel Yeh, Sean Du, Hamed Hassani, Paul Bogdan, Dawn Song, Sharon Li
| Challenge: | Uncertainty quantification (UQ) for large language models is a key building block for daily applications. |
| Approach: | They propose a general formulation of agent UQ that subsumes broad classes of existing UQ setups. |
| Outcome: | The proposed framework is based on the first general formulation of agent UQ that subsumes broad classes of existing setups. |
Natural Language Reasoning in Large Language Models: Analysis and Evaluation (2025.findings-acl)
Copied to clipboard
| Challenge: | Argumentative reasoning presents unique challenges due to its reliance on context, implicit assumptions, and value judgments. |
| Approach: | They propose a large-scale evaluation of LLMs' unconstrained natural language reasoning capabilities . they formalise a new strategy designed to evaluate argumentative reasoning in LLM . |
| Outcome: | The proposed model performs better on a range of reasoning tasks than other models. |
IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Recent approaches to quantify uncertainty in LLMs produce short or constrained answer sets, but many real-world applications require long-form and free-form text generation. |
| Approach: | They propose a framework that leverages inter-sample consistency and intra-sampled faithfulness to quantify the uncertainty in long-form LLM outputs. |
| Outcome: | The proposed framework provides reliable measures of claim-level uncertainty and the model’s faithfulness over two widely used long-form generation datasets. |
LUQ: Long-text Uncertainty Quantification for LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on Uncertainty Quantification (UQ) predominantly targets short text generation, however, real-world applications often necessitate much longer responses. |
| Approach: | They propose a method that ensembles responses from multiple models and selects the response with the lowest uncertainty. |
| Outcome: | The proposed method outperforms baseline methods in correlating with the model’s factuality scores (negative coefficient of -0.85 observed for Gemini Pro). |
CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thought (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing uncertainty quantification methods for Large language models are primarily prompt-wise rather than response-wise, which leads to inefficiency. |
| Approach: | They propose a new approach to quantify response-wise uncertainty by integrating LLMs’ inherent reasoning capabilities through Chain-of-Thought (CoT) into the UQ process. |
| Outcome: | The proposed framework outperforms existing uncertainty quantification methods and achieves an average improvement of 5.9% AUROC compared to existing methods. |
Uncertainty Quantification for In-Context Learning of Large Language Models (2024.naacl-long)
Copied to clipboard
Chen Ling, Xujiang Zhao, Xuchao Zhang, Wei Cheng, Yanchi Liu, Yiyou Sun, Mika Oishi, Takao Osaki, Katsushi Matsuda, Jie Ji, Guangji Bai, Liang Zhao, Haifeng Chen
| Challenge: | Existing studies on in-context learning have focused on quantifying the uncertainty associated with the model's response, but they neglect the complexity of the LLM and the uniqueness of in-constitut learning. |
| Approach: | They propose a method to quantify the uncertainty associated with in-context learning and propose corresponding estimation method to quantify both types of uncertainties. |
| Outcome: | The proposed method offers an unsupervised way to understand the prediction of in-context learning in a plug-and-play fashion. |
ArgBench: Benchmarking LLMs on Computational Argumentation Tasks (2026.findings-acl)
Copied to clipboard
| Challenge: | Argumentation skills are an essential toolkit for large language models (LLMs). |
| Approach: | They propose a benchmark to evaluate the generalizability of five LLM families across 46 computational argumentation tasks. |
| Outcome: | The proposed benchmark evaluates the generalizability of five LLM families across 46 computational argumentation tasks covering mining arguments, assessing perspectives, evaluating argument quality, reasoning about arguments, and generating arguments. |
Argument Summarization and its Evaluation in the Era of Large Language Models (2025.emnlp-main)
Copied to clipboard
Moritz Altemeyer, Steffen Eger, Johannes Daxenberger, Yanran Chen, Tim Altendorf, Philipp Cimiano, Benjamin Schiller
| Challenge: | Large Language Models (LLMs) have revolutionized various Natural Language Generation tasks, including Argument Summarization (ArgSum). |
| Approach: | They propose a prompt-based evaluation scheme and validate it through a human benchmark dataset. |
| Outcome: | The proposed evaluation scheme outperforms existing methods and is validated by a human benchmark dataset. |