| Challenge: | Existing evaluation models rate appropriate responses if they deviate from the ground truth . a limited range of appropriate responses for a given context is needed to evaluate dialogue . |
| Approach: | They propose an automatic evaluation model that considers similarity of generated responses with conversational context and learns parameters from an unlabeled conversation corpus. |
| Outcome: | The proposed model outperforms existing evaluation metrics in terms of correlation with human annotation scores. |
Similar Papers
Designing Precise and Robust Dialogue Response Evaluators (2020.acl-main)
Copied to clipboard
| Challenge: | Existing automated dialogue response evaluators have only moderate correlation with human judgement and are not robust. |
| Approach: | They propose to build a reference-free dialogue response evaluator that exploits the power of semi-supervised training and pretrained (masked) language models. |
| Outcome: | The proposed model achieves strong correlation with human judgement and generalizes robustly to diverse responses and corpora. |
Variational Hierarchical User-based Conversation Model (D19-1)
Copied to clipboard
| Challenge: | Recent approaches to conversation response generation model speakers and utterances together but are too tailored to the speakers. |
| Approach: | They propose a new conversation model with a stochastic variable conditioned on the speakers and affects the context. |
| Outcome: | The proposed model outperforms existing models in generating appropriate conversation responses. |
Evaluating Dialogue Generation Systems via Response Selection (2020.acl-main)
Copied to clipboard
| Challenge: | Existing automatic evaluation metrics for open-domain dialogue systems correlate poorly with human evaluation. |
| Approach: | They propose to construct response selection test sets with well-chosen false candidates to evaluate response generation systems via response selection. |
| Outcome: | The proposed method correlates with human evaluation better than widely used metrics such as BLEU. |
A Dynamic Speaker Model for Conversational Interactions (N19-1)
Copied to clipboard
| Challenge: | a neural model for characterizing individual differences in speakers is shown to be useful in human-computer interaction and dialog act prediction. |
| Approach: | They propose a neural model for learning a dynamically updated speaker embedding in a conversational context. |
| Outcome: | The proposed model is used for content ranking and dialog act prediction in human-human conversations. |
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing metrics for dialog evaluation are trained on human annotations, which is cumbersome to collect. |
| Approach: | They propose to use user sentiment and other information as proxy to measure the quality of previous dialogs. |
| Outcome: | The proposed model is comparable to models trained on human annotated data. |
Learning an Unreferenced Metric for Online Dialogue Evaluation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing tools for dialogue evaluation do not generalize to unseen datasets and/or need a human-generated reference response during inference. |
| Approach: | They propose an unreferenced automated dialogue evaluation metric that uses large pre-trained language models to extract latent representations of utterances and leverages the temporal transitions that exist between them. |
| Outcome: | The proposed model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference. |
Measuring and Improving Semantic Diversity of Dialogue Generation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation metrics for response diversity do not capture the semantic diversity of generated responses. |
| Approach: | They propose an automatic evaluation metric to measure the semantic diversity of generated responses . they show that it captures human judgments better than existing diversity metrics . |
| Outcome: | The proposed metric captures human judgments on response diversity better than existing lexical diversity metrics. |
Context-Sensitive Generation of Open-Domain Conversational Responses (C18-1)
Copied to clipboard
| Challenge: | Existing studies on single-turn conversation generation focus on coherence and context-sensitive generation of open-domain conversational responses. |
| Approach: | They propose static and dynamic attention based approaches for context-sensitive generation of open-domain conversational responses. |
| Outcome: | The proposed model outperforms all baselines on automatic and human evaluation on two public datasets. |
SelF-Eval: Self-supervised Fine-grained Dialogue Evaluation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics are expensive and easy to conduct but ineffective to reflect dialogue quality. |
| Approach: | They propose a self-supervised fine-grained dialogue evaluation framework which can automatically assign fine-granular scores for arbitrarily dialogue data. |
| Outcome: | The proposed framework is highly consistent with human evaluations and better than the state-of-the-art models. |
Generating Negative Samples by Manipulating Golden Responses for Unsupervised Learning of a Response Evaluation Model (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing metrics that rely on comparisons to a set of known correct responses do not account for the variety of responses and therefore correlate poorly with human judgment. |
| Approach: | They propose a method of manipulating a golden response to create a new negative response that is designed to be inappropriate within the context while maintaining high similarity with the original golden response. |
| Outcome: | The proposed model can be made using unsupervised learning for the next-utterance prediction task on English datasets and shows that using the negative samples alongside random negative samples can increase the model’s correlation with human evaluations. |