Mismatch between Multi-turn Dialogue and its Evaluation Metric in Dialogue State Tracking (2022.acl-short)
Copied to clipboard
| Challenge: | Existing evaluation metrics for dialog state tracking are limited for belief states accumulated as dialog proceeds . relative slot accuracy allows intuitive evaluation by assigning relative scores according to the turn of each dialog . |
| Approach: | They propose to use relative slot accuracy to complement existing evaluation metrics . joint goal accuracy and slot accuracy are used to evaluate accumulated belief states . |
| Outcome: | The proposed metrics focus on "penalizing states that fail to predict," not "reward for well-predicted states" the proposed metrics do not depend on the number of predefined slots, and allow intuitive evaluation . |
Similar Papers
Towards Fair Evaluation of Dialogue State Tracking by Flexible Incorporation of Turn-level Performances (2022.acl-short)
Copied to clipboard
| Challenge: | Dialogue State Tracking (DST) is a task-oriented conversational agent that keeps track of key information exchanged during a conversation. |
| Approach: | They propose a new evaluation metric called Flexible Goal Accuracy to address shortcomings of JGA. |
| Outcome: | The proposed metric improves on existing metrics and improves performance of turn-level and non-cumulative belief state models. |
Dual Slot Selector via Local Reliability Verification for Dialogue State Tracking (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to predict dialogue state from scratch are inefficient and lead to errors . empirical results show that our method achieves 56.93%, 60.73%, and 58.04% joint accuracy on multi-domain conversations . |
| Approach: | They propose a dual-stage dialogue state tracking method that uses a slot selector and a Slot Value generator to predict the current dialogue state. |
| Outcome: | The proposed method achieves 56.93%, 60.73%, and 58.04% joint accuracy on multi-domain conversations. |
Correctable-DST: Mitigating Historical Context Mismatch between Training and Inference for Improved Dialogue State Tracking (2022.emnlp-main)
Copied to clipboard
Hongyan Xie, Haoxiang Su, Shuangyong Song, Hao Huang, Bo Zou, Kun Deng, Jianghua Lin, Zhihui Zhang, Xiaodong He
| Challenge: | Existing dialogue state tracking approaches predict the dialogue state of a target turn sequentially based on the ground-truth previous dialogue state. |
| Approach: | They propose a method that predicts dialogue state sequentially based on previous dialogue state . they propose generating a previously “predicted” dialogue state using ground-truth previous dialogue states . |
| Outcome: | The proposed method achieves 67.51%, 68.24%, 70.30%, 71.38%, and 81.27% joint goal accuracy on MultiWOZ 2.0-2.4 datasets. |
LUNA: Learning Slot-Turn Alignment for Dialogue State Tracking (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods exploit the utterances of all dialogue turns to assign value to slots . this can lead to suboptimal results due to information introduced from irrelevant utterrances . |
| Approach: | They propose a SLot-TUrN Alignment enhanced approach to assign slot value . they explicitly align each slot with its most relevant utterance and then predict the corresponding value based on this aligned utteration. |
| Outcome: | The proposed approach achieves state-of-the-art on three multi-domain task-oriented dialogue datasets. |
Dialogue State Tracking with Explicit Slot Connection Modeling (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods to track dialogue state are lacking in multi-domain scenarios. |
| Approach: | They propose a model that explicitly considers slot correlations across domains . they propose ellipsis and reference to express values that have been mentioned by slots from other domains. |
| Outcome: | The proposed model outperforms existing models on multi-domain datasets and achieves state-of-the-art performance. |
Beyond the Granularity: Multi-Perspective Dialogue Collaborative Selection for Dialogue State Tracking (2022.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that task-oriented dialogue systems have attracted growing attention and achieved substantial progress. |
| Approach: | They propose a method that dynamically selects relevant dialogue contents for each slot . they retrieve turn-level utterances and evaluate their relevance to the slot from three perspectives . |
| Outcome: | The proposed method achieves state-of-the-art performance on MultiWOZ 2.1 and MultiWOz 2.2 and superior performance on multiple mainstream benchmark datasets. |
Improving Dialogue State Tracking with Turn-based Loss Function and Sequential Data Augmentation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models rely on a traditional cross-entropy loss function during training, which may not be optimal for improving the joint goal accuracy. |
| Approach: | They propose a Turn-based Loss Function that penalises the model if it inaccurately predicts a slot value at the early turns more so than in later turns to improve joint goal accuracy. |
| Outcome: | The proposed techniques improve the state-of-the-art model by approximately 7-8% relative reduction in error and achieve a new state- of-the art joint goal accuracy with 59.50 and 54.90 on MultiWOZ2.1 and MultiWOz2.2, respectively. |
Comprehensive Study: How the Context Information of Different Granularity Affects Dialogue State Tracking? (2021.acl-long)
Copied to clipboard
| Challenge: | Dialogue state tracking (DST) plays a key role in task-oriented dialogue systems to monitor the user’s goal. |
| Approach: | They propose to use scratch-based and previous-based strategies to track dialogue state . they explore how different granularities affect dialogue state tracking . |
| Outcome: | The scratch-based strategy obtains each slot value by inquiring all the dialogue history, while the previous-based method is not very useful for long-dependency dialogue state tracking. |
Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking (2024.lrec-main)
Copied to clipboard
| Challenge: | Current metrics for evaluating Dialogue State Tracking (DST) systems exhibit three primary limitations: i) erroneously presume a uniform distribution of slots throughout the dialog; ii) neglect to assign partial scores for individual turns; c) frequently overestimate or underestimate performance by repeatedly counting the models’ successful or failed predictions. |
| Approach: | They propose a new metric: Granular Change Accuracy (GCA) which evaluates the predicted changes in dialogue state over the entire dialogue history. |
| Outcome: | The proposed metric reduces biases arising from distribution uniformity and the positioning of errors across turns, resulting in a more precise evaluation. |
MoNET: Tackle State Momentum via Noise-Enhanced Training for Dialogue State Tracking (2023.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that MoNET outperforms previous DST methods in alleviating state momentum issues and improving the anti-noise ability. |
| Approach: | They propose to use previous state of each turn in training data as input to learn to predict current state. |
| Outcome: | The proposed model outperforms existing methods on multiWOZ datasets and shows that it can update and correct slot values and improve anti-noise ability. |