Uncertainty Determines the Adequacy of the Mode and the Tractability of Decoding in Sequence-to-Sequence Models (2022.acl-long)
Copied to clipboard
| Challenge: | In many natural language processing tasks the same input can have multiple possible outputs. |
| Approach: | They propose a novel exact n-best search algorithm for neural sequence models to measure sentence-level uncertainty by computing the degree of overlap between references from two different NLP tasks. |
| Outcome: | The proposed algorithm overly spreads the probability mass for uncertain tasks and sentences. |
Similar Papers
Uncertainty-Aware Semantic Augmentation for Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for neural machine translation only observe one source sentence at training time . this discrepancy in data distribution leads to a formidable learning challenge . |
| Approach: | They propose an uncertainty-aware semantic augmentation approach to capture universal semantic information among multiple source sentences and enhance hidden representations with this information. |
| Outcome: | The proposed approach outperforms baseline and existing methods on translation tasks. |
Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Despite their unprecedented capabilities, large language models (LLMs) often output erroneous information, which may lead users to overly rely on their false output. |
| Approach: | They formalize faithful response uncertainty based on the gap between the model’s intrinsic confidence in the assertions it makes and the decisiveness by which they are conveyed. |
| Outcome: | The proposed model is poor at faithfully conveying uncertainty on knowledge-intensive questions. |
What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability (2023.emnlp-main)
Copied to clipboard
| Challenge: | In Natural Language Generation tasks, multiple communicative goals are plausible and any goal can be put into words, or produced, in multiple ways. |
| Approach: | They characterise the extent to which human production varies lexically, syntactically, and semantically across four NLG tasks, connecting human production variability to aleatoric or data uncertainty. |
| Outcome: | The proposed model can be calibrated to human production variability using multiple samples and, when possible, multiple references. |
Your Model is Overconfident, and Other Lies We Tell Ourselves (2025.acl-long)
Copied to clipboard
| Challenge: | Analyzing 29 models, we find that difficulty is not linear or monotonic. |
| Approach: | They examine the interplay and divergence among various metrics for assessing intrinsic difficulty, including annotator dissensus, training dynamics, and model confidence. |
| Outcome: | The proposed model is based on 29 models on three datasets and analyzed by a linguistics team. |
Decoding Uncertainty: The Impact of Decoding Strategies for Uncertainty Estimation in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Decoding strategies affect the probability distribution underlying the output of a language model and can therefore affect both generation quality and uncertainty. |
| Approach: | They investigate the impact of decoding strategies on uncertainty estimation in large language models . |
| Outcome: | The proposed methods improve the uncertainty estimation of large language models by reducing repetition. |
SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers (2022.coling-1)
Copied to clipboard
Bowen Qin, Lihan Wang, Binyuan Hui, Bowen Li, Xiangpeng Wei, Binhua Li, Fei Huang, Luo Si, Min Yang, Yongbin Li
| Challenge: | Existing methods that learn from multiple semantically-equivalent questions are limited to one-to-one mapping . |
| Approach: | They propose a constraint to explore the underlying complementary semantic information among multiple semantically-equivalent questions and learn robust feature representations with reduced spurious associations. |
| Outcome: | The proposed method outperforms strong competitors and achieves state-of-the-art results on five benchmark datasets. |
Measuring Uncertainty in Neural Machine Translation with Similarity-Sensitive Entropy (2024.eacl-long)
Copied to clipboard
| Challenge: | Uncertainty estimation is an important diagnostic tool for statistical models. |
| Approach: | They propose to adapt similarity-sensitive Shannon entropy (S3E) for NMT by incorporating a concept borrowed from theoretical ecology. |
| Outcome: | The proposed framework improves quality estimation and named entity recall, and improves translation quality. |
AdaNSP: Uncertainty-driven Adaptive Decoding in Neural Semantic Parsing (P19-1)
Copied to clipboard
| Challenge: | Semantic parsing (SP) maps a natural language utterance into a formal language . standard Seq2Seq models ignore underlying grammars and may give ill-formed results. |
| Approach: | They propose an end-to-end model for semantic parsing that transduces a natural language sentence to the formal semantic representation. |
| Outcome: | The proposed model outperforms the state-of-the-art models and does not need expertise like predefined grammar or sketches in the meantime. |
Searching for Search Errors in Neural Morphological Inflection (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence models are the predominant choice for language generation tasks. |
| Approach: | They find that on word-level tasks, the empty string is often the global optimum . they suggest that the poor calibration of many neural models may stem from characteristics of a specific subset of tasks rather than general ill-suitedness of such models for language generation. |
| Outcome: | The results suggest that the poor calibration of many neural models may stem from characteristics of a specific subset of tasks rather than general ill-suitedness of such models for language generation. |
Uncertainty Unveiled: Can Exposure to More In-context Examples Mitigate Uncertainty for Large Language Models? (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in handling long sequences have unlocked new possibilities for long-context in-contact learning (ICL). |
| Approach: | They investigate how increased examples influence predictive uncertainty . they quantify uncertainty across different “shot” configurations and focus on EU . |
| Outcome: | The proposed model reduces uncertainty in simple and complex tasks by injecting task-specific knowledge. |