Hybrid Uncertainty Quantification for Selective Text Classification in Ambiguous Tasks (2023.acl-long)
Copied to clipboard
Artem Vazhentsev, Gleb Kuzmin, Akim Tsvigun, Alexander Panchenko, Maxim Panov, Mikhail Burtsev, Artem Shelmanov
| Challenge: | Existing methods for text classification tasks are inherently ambiguous and can cause errors. |
| Approach: | They propose a method that combines epistemic and aleatoric uncertainty to estimate toxicity detection errors. |
| Outcome: | The proposed method outperforms existing methods for toxicity detection and other ambiguous text classification tasks. |
Similar Papers
Uncertainty Estimation of Transformer Predictions for Misclassification Detection (2022.acl-long)
Copied to clipboard
Artem Vazhentsev, Gleb Kuzmin, Artem Shelmanov, Akim Tsvigun, Evgenii Tsymbalov, Kirill Fedyanin, Maxim Panov, Alexander Panchenko, Gleb Gusev, Mikhail Burtsev, Manvel Avetisian, Leonid Zhukov
| Challenge: | Uncertainty estimation (UE) of model predictions is crucial step for a variety of tasks such as active learning, misclassification detection, adversarial attack detection, etc. |
| Approach: | They propose to modify UE methods for Transformer models for misclassification detection in named entity recognition and text classification tasks to improve model expressiveness and computational performance. |
| Outcome: | The proposed methods outperform computationally intensive methods on misclassification detection tasks and are based on a large dataset of simulated datasets. |
How Ambiguous Are the Rationales for Natural Language Reasoning? A Simple Approach to Handling Rationale Uncertainty (2025.coling-main)
Copied to clipboard
| Challenge: | Language models have made significant progress on complex reasoning tasks, but it is impossible to obtain perfect rationales from models or even from humans. |
| Approach: | They propose a way to guide models to choose between two different reasoning paths depending on the ambiguity of rationales. |
| Outcome: | The proposed approach leads to robust performance in adversarial scenarios where rationale quality is inconsistent. |
SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Large language models have a tendency to make confidently wrong predictions, highlighting the need for uncertainty quantification (UQ) . previous studies focused on aleatoric uncertainty, but the full spectrum of uncertainties, including epistemic, remains inadequately explored. |
| Approach: | They propose a method to quantify uncertainty in large language models (LLMs) they use a set of perturbations and an aggregation module to generalize the method. |
| Outcome: | The proposed method improves model uncertainty calibration and reduces expected calibration error by 50% on average. |
Towards More Accurate Uncertainty Estimation In Text Classification (2020.emnlp-main)
Copied to clipboard
Jianfeng He, Xuchao Zhang, Shuo Lei, Zhiqian Chen, Fanglan Chen, Abdulaziz Alhamadani, Bei Xiao, ChangTien Lu
| Challenge: | Existing models of uncertainty score depend on winning score, which is the maximum probability in a semantic vector. |
| Approach: | They propose to generate accurate uncertainty score by improving the confidence of winning scores by reducing the effect of overconfidence of winning score and considering the impact of different categories simultaneously. |
| Outcome: | The proposed model reduces the effect of overconfidence of winning score and considers impact of different categories of uncertainty simultaneously. |
Uncertainty Quantification for Large Language Models (2025.acl-tutorials)
Copied to clipboard
| Challenge: | Large language models (LLMs) produce hallucinations, which undermine user trust and reliability. |
| Approach: | This tutorial offers the first systematic introduction to uncertainty quantification (UQ) for LLMs in text generation tasks. |
| Outcome: | The proposed framework provides tools for communicating the reliability of a model answer. |
Know What I don’t Know: Handling Ambiguous and Unknown Questions for Text-to-SQL (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing text-to-SQL parsers generate a plausible SQL query for arbitrary user questions, thereby failing to handle problematic user questions. |
| Approach: | They propose a weakly supervised DTE model for error detection, localization, and explanation. |
| Outcome: | The proposed model achieves the best result on real-world examples and generated examples compared with baselines. |
Trick or Neat: Adversarial Ambiguity and Language Model Evaluation (2025.findings-acl)
Copied to clipboard
| Challenge: | Direct prompting fails to detect ambiguity while linear probes can decode ambiguities with high accuracy, sometimes exceeding 90%. |
| Approach: | They introduce an adversarial ambiguity dataset that includes syntactic, lexical, and phonological ambiguities along with adversarials. |
| Outcome: | The proposed dataset includes syntactic, lexical, and phonological ambiguities along with adversarial variations. |
Aligning Language Models to Explicitly Handle Ambiguity (2024.emnlp-main)
Copied to clipboard
Hyuhng Joon Kim, Youna Kim, Cheonbok Park, Junyeob Kim, Choonghyun Park, Kang Min Yoo, Sang-goo Lee, Taeuk Kim
| Challenge: | Large language models (LLMs) are not specifically trained to deal with ambiguous utterances . ambiguity can lead to varying interpretations of the same input based on different assumptions or background knowledge . |
| Approach: | They propose a pipeline that aligns large language models to manage ambiguous queries . they propose to use their own assessment of perceived ambiguity to detect and manage queries a . |
| Outcome: | Experimental results show that APA empowers LLMs to detect and manage ambiguous queries while retaining the ability to answer clear questions. |
Handling Ambiguity in Emotion: From Out-of-Domain Detection to Distribution Estimation (2024.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that incorporating utterances without majority-agreed labels into an additional class reduces the classification performance of the other emotion classes. |
| Approach: | They propose to combine utterances without majority-agreed labels into an additional class . they propose to quantify uncertainty in emotion classification using evidential deep learning . |
| Outcome: | The proposed method retains classification accuracy while effectively detects ambiguous emotion expressions. |
Learning Disentangled Representations of Negation and Uncertainty (2022.acl-long)
Copied to clipboard
| Challenge: | Negation and uncertainty modeling are long-standing tasks in natural language processing. |
| Approach: | They propose to disentangle negation, uncertainty, and content using a Variational Autoencoder by supervising latent representations using auxiliary objectives. |
| Outcome: | The proposed model can disentangle negation, uncertainty, and content using a Variational Autoencoder. |