Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance (2021.emnlp-main)
Copied to clipboard
Carel van Niekerk, Andrey Malinin, Christian Geishauser, Michael Heck, Hsien-chin Lin, Nurul Lubis, Shutong Feng, Milica Gasic
| Challenge: | Neural dialogue belief trackers that take uncertainty into account are often overconfident in their decisions and therefore less robust. |
| Approach: | They propose to use different uncertainty measures in neural belief tracking to integrate uncertainty into the feature space of the policy and train policies through interaction with a user simulator. |
| Outcome: | The proposed approach improves both performance and robustness of the downstream dialogue policy. |
Similar Papers
Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensembles (2020.findings-emnlp)
Copied to clipboard
Carel van Niekerk, Michael Heck, Christian Geishauser, Hsien-chin Lin, Nurul Lubis, Marco Moresi, Milica Gasic
| Challenge: | Current models for dialogue state tracking only achieve 55% accuracy . however, they lack in performance compared to belief trackers and do not produce well calibrated distributions. |
| Approach: | They propose to calibrate a model for dialogue belief trackers to measure dialogue state accuracy. |
| Outcome: | The proposed model outperforms existing models in terms of accuracy and accuracy. |
Fully Statistical Neural Belief Tracking (P18-2)
Copied to clipboard
| Challenge: | Existing framework for a dialogue state tracking model requires an expensive manual retuning step . |
| Approach: | They propose to improve existing NBT model by removing a manual retuning step . they propose two different statistical update mechanisms to improve model performance . |
| Outcome: | The proposed model achieves competitive performance and provides a robust framework for building resource-light DST models. |
Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Effective interlocutors account for the uncertain goals, beliefs, and emotions of others. |
| Approach: | They propose to calibrate language models to better represent outcome uncertainty . they propose to use two methods to calibrated small open-source models . |
| Outcome: | The proposed fine-tuning strategies can calibrate smaller open-source models to beat pre-trained models 10x their size. |
“None of the Above”: Measure Uncertainty in Dialog Response Retrieval (2020.acl-main)
Copied to clipboard
| Challenge: | End-to-end (E2E) dialog retrieval models jointly encode a dialog and a candidate response, assuming the ground truth is always present in the candidate set. |
| Approach: | They propose to capture the original retrieval model's confidence concerning the best prediction using trivial additional computation. |
| Outcome: | The proposed model can capture the model's confidence concerning the best prediction using trivial additional computation. |
Investigating the Impact of Model Instability on Explanations and Uncertainty (2024.findings-acl)
Copied to clipboard
| Challenge: | Explainable AI methods are typically evaluated holistically, but small perturbations to inputs can vastly distort explanations. |
| Approach: | They artificially simulate epistemic uncertainty in text input by introducing noise at inference time and measure the effect on the output of pre-trained language models. |
| Outcome: | The proposed model can detect salient tokens when uncertain, but it is not reliable when small perturbations are exposed during training. |
Dialogue State Tracking with Incremental Reasoning (2021.tacl-1)
Copied to clipboard
| Challenge: | Empirical results show that our method outperforms the state-of-the-art methods in terms of joint belief accuracy. |
| Approach: | They propose to track dialogue states gradually with reasoning over dialogue turns using the back-end data. |
| Outcome: | Empirical results show that the proposed method outperforms state-of-the-art methods in terms of joint belief accuracy for a large-scale human–human dialogue dataset. |
On the Interaction of Belief Bias and Explanations (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to evaluate explainability fail to account for belief biases affecting human performance . previous studies have shown that neural models can make confident predictions relying on artifacts . |
| Approach: | They propose to account for belief bias in explainability by using models of varying quality and adversarial examples. |
| Outcome: | The proposed methods show that results change when using models of varying quality and adversarial examples. |
On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Study (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Modern deep models for summarization generate miscalibrated predictive uncertainty, compromising reliability and trustworthiness in real-world applications. |
| Approach: | They propose to use probabilistic methods to improve the uncertainty quality of neural summarization models by using three large-scale benchmarks with varying difficulty. |
| Outcome: | The proposed methods consistently improve the model’s generation and uncertainty quality, leading to improved selective generation performance (i.e., abstaining from low-quality summaries) in practice. |
Relying on the Unreliable: The Impact of Language Models’ Reluctance to Express Uncertainty (2024.acl-long)
Copied to clipboard
| Challenge: | a pivotal aspect of fostering reliable human-AI interactions lies in the apt communication of model confidences. |
| Approach: | They examine how LMs incorporate confidence in responses via natural language . they also examine how downstream users behave in response to LM-articulated uncertainties . |
| Outcome: | The proposed model overconfidences are high in LMs, and humans are biased against uncertainty-rich texts. |
Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features (2021.acl-long)
Copied to clipboard
| Challenge: | Existing systems that strive to be informative teachers are difficult to build . knowledge grounded dialogue systems are difficult because of limited training objectives . |
| Approach: | They propose to train a generative neural dialogue model that is controlled to stay faithful to evidence . they propose to use additional inputs to generate more objective responses . |
| Outcome: | The proposed model produces responses that are perceived by humans to be objective and faithful to evidence. |