Learning the Human Judgment for the Automatic Evaluation of Chatbot (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing evaluation methods for dialogue systems rely on human judges to label quality of generated text. |
| Approach: | They propose a machine learning approach to reduce the effort of human evaluation by learning the human judgment on comparing two generative dialogue systems. |
| Outcome: | The proposed method reduces the effort of human evaluation by learning which generative models is better in each dialog context. |
Similar Papers
Designing Precise and Robust Dialogue Response Evaluators (2020.acl-main)
Copied to clipboard
| Challenge: | Existing automated dialogue response evaluators have only moderate correlation with human judgement and are not robust. |
| Approach: | They propose to build a reference-free dialogue response evaluator that exploits the power of semi-supervised training and pretrained (masked) language models. |
| Outcome: | The proposed model achieves strong correlation with human judgement and generalizes robustly to diverse responses and corpora. |
Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems (2020.emnlp-main)
Copied to clipboard
Jan Deriu, Don Tuggener, Pius von Däniken, Jon Ander Campos, Alvaro Rodrigo, Thiziri Belkacem, Aitor Soroa, Eneko Agirre, Mark Cieliebak
| Challenge: | Lack of time efficient and reliable evalu-ation methods is hampering the development of conversational dialogue systems (chatbots). |
| Approach: | They propose a framework that replaces human-bot conversations with conversations between bots and an annotation tool that ranks chatbots based on their ability to mimic human behaviour. |
| Outcome: | The proposed evaluation framework replaces human-bot conversations with bot conversations and allows for frequent evaluations of chatbots during their evaluation cycle. |
Automating Human Evaluation of Dialogue Systems (2022.naacl-srw)
Copied to clipboard
| Challenge: | a recent study shows that human evaluations of dialogue systems weakly reflect human judgments. |
| Approach: | They propose a BERT-based model that fine-tunes a model with three prediction heads to predict whether the system-generated output is natural, fluent, and informative. |
| Outcome: | The proposed model achieves an average accuracy of 77% over the 3 labels . it also uses three different models to compute the labels compared to three separate models . |
Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation (D19-1)
Copied to clipboard
| Challenge: | Existing evaluation methods for natural language generation are inadequate . distinguishing machine-generated text is challenging even for human evaluators . |
| Approach: | They compare human-based evaluators with automated evaluation procedures . they find human evaluers do not correlate well with discriminative evalators . |
| Outcome: | The proposed evaluation methods are compared with a dozen state-of-the-art generators for online product reviews. |
Automatic Evaluate Dialogue Appropriateness by Using Dialogue Act (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluations of dialogue quality rely on human judgments, which are time-consuming, labor-intensive, prone to biases, and lacking objectivity. |
| Approach: | They propose a method that utilizes the underlying patterns of dialogue act transitions to evaluate the appropriateness of chatbot responses. |
| Outcome: | The proposed method proves that human judgments are time-consuming, labor-intensive, and lacking objectivity. |
ChatEval: A Tool for Chatbot Evaluation (N19-4)
Copied to clipboard
| Challenge: | open-domain dialog systems are difficult to evaluate due to lack of standardization and standardization in evaluation procedures. |
| Approach: | They propose a framework for human evaluation of chatbots that augments existing tools . researchers can submit their trained models to the ChatEval web interface . reproducibility and model assessment for opendomain dialog systems is challenging . |
| Outcome: | The proposed framework provides a web-based hub for researchers to compare their models with baselines and prior work. |
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing metrics for dialog evaluation are trained on human annotations, which is cumbersome to collect. |
| Approach: | They propose to use user sentiment and other information as proxy to measure the quality of previous dialogs. |
| Outcome: | The proposed model is comparable to models trained on human annotated data. |
Achieving Reliable Human Assessment of Open-Domain Dialogue Systems (2022.acl-long)
Copied to clipboard
| Challenge: | Evaluation of open-domain dialogue systems is challenging and unreliable . human evaluation of live conversations is highly reliable, but reliability cannot be assumed . |
| Approach: | They propose a method of open-domain dialogue evaluation that is highly reliable . they compare live conversations with models that avoid pre-created reference dialogues . |
| Outcome: | The proposed method is highly reliable while remaining feasible and low cost. |
Exploring the Impact of Human Evaluator Group on Chat-Oriented Dialogue Evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | Evaluator groups such as domain experts, university students, and crowdworkers have been used to assess and compare chat-oriented dialogue systems. |
| Approach: | They analyze the impact of evaluator groups on dialogue system evaluation by testing 4 state-of-the-art dialogue systems using 4 distinct evaluer groups. |
| Outcome: | The proposed evaluations show that the evaluator group impact is not seen for Pairwise, and that it is beneficial for certain metrics. |
You Impress Me: Dialogue Generation via Mutual Persona Perception (2020.acl-main)
Copied to clipboard
| Challenge: | Existing chit-chat systems tend to generate uninformative responses and lack coherent personality traits due to the diversity of speakers. |
| Approach: | They propose a transmitter-receiver framework which explicitly models understanding between interlocutors. |
| Outcome: | The proposed framework improves on a large public dataset, Persona-Chat, with a significant boost over the state-of-the-art frameworks. |