USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Standard language generation metrics have been shown to be ineffective for dialog evaluation. |
| Approach: | They propose an unsupervised evaluation metric for dialog that trains unsupervised models to measure several desirable qualities of dialog. |
| Outcome: | The proposed evaluation metric strongly correlates with human judgment on Topical-Chat and PersonaChat. |
Similar Papers
Learning an Unreferenced Metric for Online Dialogue Evaluation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing tools for dialogue evaluation do not generalize to unseen datasets and/or need a human-generated reference response during inference. |
| Approach: | They propose an unreferenced automated dialogue evaluation metric that uses large pre-trained language models to extract latent representations of utterances and leverages the temporal transitions that exist between them. |
| Outcome: | The proposed model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference. |
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing metrics for dialog evaluation are trained on human annotations, which is cumbersome to collect. |
| Approach: | They propose to use user sentiment and other information as proxy to measure the quality of previous dialogs. |
| Outcome: | The proposed model is comparable to models trained on human annotated data. |
REAM♯: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics for open-domain dialogue systems are limited by the diversity of possible outcomings. |
| Approach: | They propose a method to augment a reference set to improve reliability . they propose BLEU to measure similarity between a predicted response and a small set of references . |
| Outcome: | The proposed model improves the reliability of reference-based metrics with augmented reference sets. |
xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, human evaluation is the most reliable way to holistically judge the quality of the dialogue. |
| Approach: | They propose to use English dialogue evaluation metrics to generalize them to other languages. |
| Outcome: | The proposed metrics outperform OpenAI’s ChatGPT in terms of average Pearson correlations over all datasets and languages. |
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent model-based reference-free metrics for open-domain dialogue evaluation lack correlations with human judgment and poor interpretability. |
| Approach: | They propose a multi-dimensional dialogue-level metric with three sub-metrics targeting a specific dimension. |
| Outcome: | The proposed metric outperforms existing models and sub-metrics in three high-quality dialogue evaluation benchmarks. |
Spurious Correlations in Reference-Free Evaluation of Text Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work suggests that reference-free evaluation metrics may rely on spurious correlations with human judgments. |
| Approach: | They propose to use model-based, reference-free evaluation metrics to evaluate natural language generation systems. |
| Outcome: | The proposed metrics achieve high correlations with human judgments, but they may not be robust enough to evaluate their efficacy and robustness. |
Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems (2020.coling-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics are not designed to cope with this flexibility. |
| Approach: | They propose to group the qualities into three groups to obtain a single metric called USL-H. |
| Outcome: | The proposed metric achieves good correlations with human judgment and maintains its configurability towards different aspects and metrics. |
Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing automatic dialog evaluation metrics are mostly reference-based . Existing models that measure self-reported user ratings are biased and variance among different users. |
| Approach: | They propose an automatic evaluation model that automatically cleans self-reported user ratings as it trains on them. |
| Outcome: | The proposed model achieves 89.2% accuracy in the dialog comparison task. |
CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systems (2025.coling-main)
Copied to clipboard
| Challenge: | Existing metrics for dialogue quality evaluation show low correlation with human judgements . current metrics do not accurately evaluate dialogue responses based on dialogue history . |
| Approach: | They propose a new metric measuring causal strength between dialogue histories and responses . they collect a dialogue dataset with human-annotated causal relations and pairwise human judgements . |
| Outcome: | The proposed metric outperforms existing state-of-the-art metrics in human judgements . it is based on a dialogue dataset with human-annotated causal relations and human judgement sets . |
uBLEU: Uncertainty-Aware Automatic Evaluation Method for Open-Domain Dialogue Systems (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation metrics for text generation tasks do not consider uncertain responses without writing additional reference responses by hand. |
| Approach: | They propose a human-aided, uncertainty-aware evaluation method for open-domain dialogue systems, BLEU. |
| Outcome: | The proposed method is comparable to existing methods on Twitter and improves state-of-the-art evaluation method RUBER. |