| Challenge: | Existing models that measure engagement use expensive human annotas and abstract definitions of the term. |
| Approach: | They propose a human-reaction based model to evaluate dialogue engagingness . they propose combining distant-supervision with a theoretical foundation for engagement . |
| Outcome: | The proposed model is trained on 80k Reddit-based engagement datasets . it uses distant-supervision from human-reaction feedback to evaluate dialogue engagementness . |
Similar Papers
Training Millions of Personalized Dialogue Agents (D18-1)
Copied to clipboard
| Challenge: | Current dialogue systems fail at being engaging for users when trained end-to-end without relying on proactive reengaging scripted strategies. |
| Approach: | They propose a dataset that provides 5 million personas and 700 million person-based dialogues. |
| Outcome: | The proposed dataset provides 5 million personas and 700 million person-based dialogues. |
MEEP: Is this Engaging? Prompting Large Language Models for Dialogue Evaluation in Multilingual Settings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing metrics for engagingness evaluate the response without the conversation history, are designed for one dataset, or have limited correlation with human annotations. |
| Approach: | They propose to use large language models to evaluate engagingness in dialogue . they propose to include prompts and translated prompts in the model . |
| Outcome: | The proposed model outperforms existing methods on evaluation of engagingness in dialogue across languages. |
Learning an Unreferenced Metric for Online Dialogue Evaluation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing tools for dialogue evaluation do not generalize to unseen datasets and/or need a human-generated reference response during inference. |
| Approach: | They propose an unreferenced automated dialogue evaluation metric that uses large pre-trained language models to extract latent representations of utterances and leverages the temporal transitions that exist between them. |
| Outcome: | The proposed model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference. |
Image-Chat: Engaging Grounded Conversations (2020.acl-main)
Copied to clipboard
| Challenge: | In order for machines to communicate with humans, they must understand the natural things that humans say about the world they live in and respond in kind. |
| Approach: | They propose to fuse a set of neural architectures using image and text representations to achieve this goal. |
| Outcome: | The proposed model performs well on the Image-Chat task and humans prefer it 47.7% of the time. |
Dialogue Response Ranking Training with Large-Scale Human Feedback Data (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing open-domain dialog models can minimize the perplexity of target human responses . however, some human responses are more engaging than others, spawning more followup interactions . |
| Approach: | They train open-domain dialog models to minimize perplexity of target human responses . they use social media feedback data to train models to predict engaging dialog turns . |
| Outcome: | The proposed model outperforms existing models on 133M human feedback pairs . it also outperformed the conventional dialog perplexity baseline model . |
RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue (2023.acl-long)
Copied to clipboard
| Challenge: | Evaluating open-domain dialogue systems is challenging because of the one-to-many problem. |
| Approach: | They propose a reference-based dialogue evaluation approach that leverages the pre-created utterance as reference other than the gold response to relieve the one-to-many problem. |
| Outcome: | The proposed method outperforms state-of-the-art evaluation methods on three datasets and two existing benchmarks. |
SelF-Eval: Self-supervised Fine-grained Dialogue Evaluation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics are expensive and easy to conduct but ineffective to reflect dialogue quality. |
| Approach: | They propose a self-supervised fine-grained dialogue evaluation framework which can automatically assign fine-granular scores for arbitrarily dialogue data. |
| Outcome: | The proposed framework is highly consistent with human evaluations and better than the state-of-the-art models. |
Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset (P19-1)
Copied to clipboard
| Challenge: | EmpatheticDialogues dataset provides a benchmark for empathetic dialogue generation . human evaluators perceive dialogue models as more epathetic . |
| Approach: | They propose a benchmark for empathetic dialogue generation from a dataset of 25k conversations grounded in emotional situations. |
| Outcome: | The proposed benchmarks show that existing models are perceived to be more empathetic by human evaluators compared to models trained on large-scale Internet conversations. |
xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, human evaluation is the most reliable way to holistically judge the quality of the dialogue. |
| Approach: | They propose to use English dialogue evaluation metrics to generalize them to other languages. |
| Outcome: | The proposed metrics outperform OpenAI’s ChatGPT in terms of average Pearson correlations over all datasets and languages. |
What Did You Refer to? Evaluating Co-References in Dialogue (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing neural end-to-end dialogue models have limitations on exactly interpreting the linguistic structures in dialogue history context. |
| Approach: | They propose to directly measure the capability of neural end-to-end dialogue models on understanding the entity-oriented structures via question answering. |
| Outcome: | The proposed model can understand large-scale English and Chinese human human dialogues using a large-format dataset. |