Joint Turn and Dialogue level User Satisfaction Estimation on Multi-Domain Conversations (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to estimate turn and dialogue level user satisfaction employ hand-crafted features and rely on complex annotation schemes, which reduce generalizability of the trained models. |
| Approach: | They propose to use an adaptive multi-task loss function to minimize hand-crafted features to estimate user satisfaction at turn level from an end user perspective. |
| Outcome: | The proposed model improves on 28 Alexa domains, two dialogue systems and three user groups on a set of user-generated dialogues from 28 Alexia domain and 28 Alexis domains. |
Similar Papers
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent model-based reference-free metrics for open-domain dialogue evaluation lack correlations with human judgment and poor interpretability. |
| Approach: | They propose a multi-dimensional dialogue-level metric with three sub-metrics targeting a specific dimension. |
| Outcome: | The proposed metric outperforms existing models and sub-metrics in three high-quality dialogue evaluation benchmarks. |
DynaEval: Unifying Turn and Dialogue Level Evaluation (2021.acl-long)
Copied to clipboard
Chen Zhang, Yiming Chen, Luis Fernando D’Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, Haizhou Li
| Challenge: | Existing evaluation metrics focus on the turn-level quality of a dialogue . a unified framework that holistically considers the quality of the entire dialogue is needed . |
| Approach: | They propose a unified automatic evaluation framework which holistically considers the quality of the entire dialogue. |
| Outcome: | The proposed framework outperforms the state-of-the-art dialogue coherence model and correlates strongly with human judgements across multiple evaluation aspects at both turn and dialogue level. |
Self-Supervised Contrastive Learning for Efficient User Satisfaction Prediction in Conversational Agents (2021.naacl-main)
Copied to clipboard
| Challenge: | End-to-end deep learning methods that focus on user satisfaction are challenging due to the required annotation costs and turnaround times. |
| Approach: | They propose a self-supervised contrastive learning approach that leverages the pool of unlabeled data to learn user-agent interactions. |
| Outcome: | The proposed approach reduces the required number of annotations while improving generalization on unseen skills. |
xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, human evaluation is the most reliable way to holistically judge the quality of the dialogue. |
| Approach: | They propose to use English dialogue evaluation metrics to generalize them to other languages. |
| Outcome: | The proposed metrics outperform OpenAI’s ChatGPT in terms of average Pearson correlations over all datasets and languages. |
MuTual: A Dataset for Multi-Turn Dialogue Reasoning (2020.acl-main)
Copied to clipboard
| Challenge: | Existing non-task oriented dialogue systems can yield a relevant and fluent response, but sometimes make logical mistakes because of weak reasoning capabilities. |
| Approach: | They propose a dataset for multi-turn dialogue reasoning that uses annotated dialogues to train a machine to handle various reasoning problems. |
| Outcome: | Empirical results show that state-of-the-art methods only reach 71%, far behind human performance of 94%. |
Joint Goal Segmentation and Goal Success Prediction on Multi-Domain Conversations (2022.coling-1)
Copied to clipboard
| Challenge: | Existing metrics to measure the performance of conversational AI assistants are difficult to establish due to their slow nature. |
| Approach: | They propose an automatic dialogue evaluation framework that performs goal segmentation and success prediction by adding multi-task learning heads. |
| Outcome: | The proposed model achieves on-par with human annotation compared to a gold annotation benchmark. |
SelF-Eval: Self-supervised Fine-grained Dialogue Evaluation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics are expensive and easy to conduct but ineffective to reflect dialogue quality. |
| Approach: | They propose a self-supervised fine-grained dialogue evaluation framework which can automatically assign fine-granular scores for arbitrarily dialogue data. |
| Outcome: | The proposed framework is highly consistent with human evaluations and better than the state-of-the-art models. |
Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogs (2023.acl-industry)
Copied to clipboard
Abishek Komma, Nagesh Panyam Chandrasekarasastry, Timothy Leffel, Anuj Goyal, Angeliki Metallinou, Spyros Matsoukas, Aram Galstyan
| Challenge: | Existing methods to dialog quality estimation focus on evaluating individual turns or collecting dialog-level quality measurements from end users immediately following an interaction. |
| Approach: | They propose a dialog-level annotation workflow called Dialog Quality Annotation . they propose to annotate dialogs for attributes such as goal completion and user sentiment . |
| Outcome: | The proposed model outperforms existing methods for dialog quality estimation . it shows that high-quality human-annotated data is important for dialog-quality evaluation . |
CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems (2024.findings-acl)
Copied to clipboard
| Challenge: | a number of studies have evaluated user satisfaction estimation in TOD systems . current benchmarks for user satisfaction estimates are highly skewed towards dialogues for which the user is satisfied. |
| Approach: | They leverage large language models to generate satisfaction-aware counterfactual dialogues to augment original dialogues of a test collection. |
| Outcome: | The proposed models show higher robustness to increase in dissatisfaction labels than fine-tuned models. |
Multi-turn Response Selection using Dialogue Dependency Relations (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for multi-turn response selection ignore the dependencies between the turns. |
| Approach: | They propose a dialogue extraction algorithm to transform a dialog history into threads based on their dependency relations. |
| Outcome: | The proposed model outperforms the state-of-the-art models on DSTC7 and DSTF8* with competitive results on UbuntuV2 . |