Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems (2020.coling-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics are not designed to cope with this flexibility. |
| Approach: | They propose to group the qualities into three groups to obtain a single metric called USL-H. |
| Outcome: | The proposed metric achieves good correlations with human judgment and maintains its configurability towards different aspects and metrics. |
Similar Papers
IM^2: an Interpretable and Multi-category Integrated Metric Framework for Automatic Dialogue Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Evaluation metrics for dialogue systems are expensive and time-consuming . current evaluation metrics focus on a single quality or several qualities . |
| Approach: | They propose an interpretable, multi-faceted, and controllable framework to combine dialogue metrics which are good at measuring different qualities. |
| Outcome: | The proposed framework integrates a large number of evaluation metrics to improve the performance of the model. |
Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods of open-domain dialogue evaluation are labor-intensive and inefficient. |
| Approach: | They propose to use open-domain dialogues to evaluate different aspects of dialogues using holistic evaluation metrics. |
| Outcome: | The proposed metrics show strong correlations with human judgments. |
REAM♯: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics for open-domain dialogue systems are limited by the diversity of possible outcomings. |
| Approach: | They propose a method to augment a reference set to improve reliability . they propose BLEU to measure similarity between a predicted response and a small set of references . |
| Outcome: | The proposed model improves the reliability of reference-based metrics with augmented reference sets. |
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent model-based reference-free metrics for open-domain dialogue evaluation lack correlations with human judgment and poor interpretability. |
| Approach: | They propose a multi-dimensional dialogue-level metric with three sub-metrics targeting a specific dimension. |
| Outcome: | The proposed metric outperforms existing models and sub-metrics in three high-quality dialogue evaluation benchmarks. |
CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systems (2025.coling-main)
Copied to clipboard
| Challenge: | Existing metrics for dialogue quality evaluation show low correlation with human judgements . current metrics do not accurately evaluate dialogue responses based on dialogue history . |
| Approach: | They propose a new metric measuring causal strength between dialogue histories and responses . they collect a dialogue dataset with human-annotated causal relations and pairwise human judgements . |
| Outcome: | The proposed metric outperforms existing state-of-the-art metrics in human judgements . it is based on a dialogue dataset with human-annotated causal relations and human judgement sets . |
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021.emnlp-main)
Copied to clipboard
| Challenge: | Evaluation metrics are a key ingredient for progress of text generation systems . a class of novel evaluation metrics based on BERT and its variants has been explored . |
| Approach: | They propose to disentangle BERT-based evaluation metrics along linguistic factors . they show they are sensitive to lexical overlap, just like BLEU and ROUGE . |
| Outcome: | The proposed metrics capture all aspects but are sensitive to lexical overlap, just like BLEU and ROUGE, the authors show . |
Learning an Unreferenced Metric for Online Dialogue Evaluation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing tools for dialogue evaluation do not generalize to unseen datasets and/or need a human-generated reference response during inference. |
| Approach: | They propose an unreferenced automated dialogue evaluation metric that uses large pre-trained language models to extract latent representations of utterances and leverages the temporal transitions that exist between them. |
| Outcome: | The proposed model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference. |
uBLEU: Uncertainty-Aware Automatic Evaluation Method for Open-Domain Dialogue Systems (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation metrics for text generation tasks do not consider uncertain responses without writing additional reference responses by hand. |
| Approach: | They propose a human-aided, uncertainty-aware evaluation method for open-domain dialogue systems, BLEU. |
| Outcome: | The proposed method is comparable to existing methods on Twitter and improves state-of-the-art evaluation method RUBER. |
Proxy Indicators for the Quality of Open-domain Dialogues (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluation of open-domain dialogues are expensive and require human annotators to evaluate their quality. |
| Approach: | They propose to use a deep-learning model trained on the general language understanding evaluation benchmark to serve as a quality indication of open-domain dialogues. |
| Outcome: | The proposed model can infer various quality metrics and derive a component-based overall score. |
GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics only consider surface features or utterance-level semantics, without explicitly considering the fine-grained topic transition dynamics of dialogue flows. |
| Approach: | They propose a graph-enhanced evaluation metric GRADE to evaluate dialogue coherence . GRADE incorporates utterance-level contextualized representations and fine-grained topic-level graph representations to improve communication logic. |
| Outcome: | The proposed evaluation metric outperforms state-of-the-art metrics on measuring diverse dialogue models in terms of Pearson and Spearman correlations with human judgments. |