Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics (2022.findings-emnlp)
Copied to clipboard
Hyundong Cho, Chinnadhurai Sankar, Christopher Lin, Kaushik Sadagopan, Shahin Shayandeh, Asli Celikyilmaz, Jonathan May, Ahmad Beirami
| Challenge: | Recent studies have revealed the vulnerability of dialogue state tracking models to distributional shifts, resulting in poor performance. |
| Approach: | They present a toolkit for standardized and comprehensive dialogue state tracking diagnoses that provides a richer summary of strengths and weaknesses. |
| Outcome: | The proposed toolkit shows that different classes of DST models have clear strengths and weaknesses, while generation models are more promising for handling language variety and span-based classification models are robust to unseen entities. |
Similar Papers
Robust Dialogue State Tracking with Weak Supervision and Sparse Data (2022.tacl-1)
Copied to clipboard
Michael Heck, Nurul Lubis, Carel van Niekerk, Shutong Feng, Christian Geishauser, Hsien-Chin Lin, Milica Gašić
| Challenge: | Generalizing dialogue state tracking (DST) to new data and domains is especially challenging due to the strong reliance on abundant and fine-grained supervision during training. |
| Approach: | They propose a training strategy to build extractive DST models without the need for fine-grained manual span labels. |
| Outcome: | The proposed model improves robustness against sample sparsity, new concepts, and topics, leading to state-of-the-art performance on a range of benchmarks. |
Disfluency Generation for More Robust Dialogue Systems (2023.findings-acl)
Copied to clipboard
| Challenge: | Disfluencies in user utterances can trigger a chain of errors impacting all the modules of a dialogue system. |
| Approach: | They propose to augment existing dialogue datasets with disfluent utterances by paraphrasing them into disfluente ones. |
| Outcome: | The proposed method improves dialogue state tracking and response generation by combining disfluent utterances with disfluency utteraces. |
Correctable-DST: Mitigating Historical Context Mismatch between Training and Inference for Improved Dialogue State Tracking (2022.emnlp-main)
Copied to clipboard
Hongyan Xie, Haoxiang Su, Shuangyong Song, Hao Huang, Bo Zou, Kun Deng, Jianghua Lin, Zhihui Zhang, Xiaodong He
| Challenge: | Existing dialogue state tracking approaches predict the dialogue state of a target turn sequentially based on the ground-truth previous dialogue state. |
| Approach: | They propose a method that predicts dialogue state sequentially based on previous dialogue state . they propose generating a previously “predicted” dialogue state using ground-truth previous dialogue states . |
| Outcome: | The proposed method achieves 67.51%, 68.24%, 70.30%, 71.38%, and 81.27% joint goal accuracy on MultiWOZ 2.0-2.4 datasets. |
Beyond Single-User Dialogue: Assessing Multi-User Dialogue State Tracking Capabilities of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have demonstrated remarkable performance in zero-shot dialogue state tracking (DST), reducing the need for task-specific training. |
| Approach: | They extend existing DST dataset by generating utterances of a second user based on speech act theory. |
| Outcome: | The proposed model incorporates utterances of a second user into conversations, enabling a controlled evaluation of LLMs in multi-user settings. |
GCDST: A Graph-based and Copy-augmented Multi-domain Dialogue State Tracking (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to training DST on a single domain ignore information across domains. |
| Approach: | They construct a dialogue state graph to transfer structured features among related domain-slot pairs across domains and encode the graph information of dialogue states by graph convolutional networks. |
| Outcome: | The proposed model improves the performance of the multi-domain DST baseline with the absolute joint accuracy of 2.0% and 1.0% on the MultiWOZ 2.0 and 2.1 dialogue datasets. |
MetaASSIST: Robust Dialogue State Tracking with Meta Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing dialogue datasets contain lots of noise in their state annotations. |
| Approach: | They propose a framework to train robust dialogue state tracking models by combining pseudo and vanilla labels by a common weighting parameter. |
| Outcome: | The proposed framework achieves state-of-the-art accuracy of 80.10% on multiWOZ 2.4. |
Towards LLM-driven Dialogue State Tracking (2023.emnlp-main)
Copied to clipboard
| Challenge: | emergence of large language models (LLMs) such as GPT3 and ChatGPT has sparked considerable interest in assessing their efficacy across diverse applications. |
| Approach: | They present a framework for a domain-slot instruction tuning method that allows LDST to achieve performance on par with ChatGPT. |
| Outcome: | The proposed framework performs better in zero-shot and few-shot settings than previous SOTA methods. |
Out-of-Task Training for Dialog State Tracking Models (2020.coling-main)
Copied to clipboard
Michael Heck, Christian Geishauser, Hsien-chin Lin, Nurul Lubis, Marco Moresi, Carel van Niekerk, Milica Gasic
| Challenge: | Dialog state tracking (DST) suffers from data sparsity. |
| Approach: | They utilize non-dialog data from unrelated NLP tasks to train dialog state trackers . they propose to use dialog state tracking to summarise the conversation history . |
| Outcome: | The proposed method exploits non-dialog data from unrelated NLP tasks to train dialog state trackers. |
Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking (2022.findings-acl)
Copied to clipboard
| Challenge: | Annotating task-oriented dialogues is notorious for the expensive and difficult data collection process. |
| Approach: | They propose to reformulate dialogue state tracking as a dialogue summarization problem by using synthetic dialogue summaries generated by a set of rules. |
| Outcome: | The proposed method outperforms previous studies on few-shot dialogue state tracking in MultiWoZ 2.0 and 2.1 in cross-domain and multi-domain settings. |
PromptAttack: Probing Dialogue State Trackers with Adversarial Prompts (2023.findings-acl)
Copied to clipboard
| Challenge: | Toward building more robust and reliable conversational systems, we introduce a prompt-based learning approach to automatically generate effective adversarial examples to probe DST models. |
| Approach: | They propose a prompt-based learning approach to automatically generate effective adversarial examples to probe DST models. |
| Outcome: | The proposed framework leads to the greatest reduction in accuracy and the best attack success rate while maintaining good fluency and a low perturbation ratio. |