ChatHF: Collecting Rich Human Feedback from Real-time Conversations (2024.emnlp-demo)
Copied to clipboard
| Challenge: | We present an interactive framework for chatbot evaluation that integrates configurable annotation within a chat interface. |
| Approach: | They propose an interactive framework for chatbot evaluation that integrates configurable annotation within a chat interface. |
| Outcome: | The proposed framework supports fine-grained error detection and human evaluation at the same time. |
Similar Papers
ChatEval: A Tool for Chatbot Evaluation (N19-4)
Copied to clipboard
| Challenge: | open-domain dialog systems are difficult to evaluate due to lack of standardization and standardization in evaluation procedures. |
| Approach: | They propose a framework for human evaluation of chatbots that augments existing tools . researchers can submit their trained models to the ChatEval web interface . reproducibility and model assessment for opendomain dialog systems is challenging . |
| Outcome: | The proposed framework provides a web-based hub for researchers to compare their models with baselines and prior work. |
Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems (2020.emnlp-main)
Copied to clipboard
Jan Deriu, Don Tuggener, Pius von Däniken, Jon Ander Campos, Alvaro Rodrigo, Thiziri Belkacem, Aitor Soroa, Eneko Agirre, Mark Cieliebak
| Challenge: | Lack of time efficient and reliable evalu-ation methods is hampering the development of conversational dialogue systems (chatbots). |
| Approach: | They propose a framework that replaces human-bot conversations with conversations between bots and an annotation tool that ranks chatbots based on their ability to mimic human behaviour. |
| Outcome: | The proposed evaluation framework replaces human-bot conversations with bot conversations and allows for frequent evaluations of chatbots during their evaluation cycle. |
Towards Boosting the Open-Domain Chatbot with Human Feedback (2023.acl-long)
Copied to clipboard
| Challenge: | Existing frameworks for pre-training open-domain dialogue models with social media comments generate coherent replies but have difficulties producing engaging responses. |
| Approach: | They propose a framework to boost the open-domain chatbot by leveraging human feedback and annotating the model's candidate responses. |
| Outcome: | The proposed framework boosts the open-domain chatbot by leveraging human demonstrated responses and leveraging the implicit preference in the data collection process. |
ChatMatch: Evaluating Chatbots by Autonomous Chat Tournaments (2022.acl-long)
Copied to clipboard
| Challenge: | Existing automated evaluation systems of chatbots rely on static chat scripts as ground truth, which is hard to obtain. |
| Approach: | They propose an interactive chatbot evaluation framework that allows chatbots to compete with each other like in a sports tournament. |
| Outcome: | The proposed framework can rank chatbots independently from their model architectures and domains . existing evaluation systems rely on static chat scripts as ground truth . |
Annobot: Platform for Annotating and Creating Datasets through Conversation with a Chatbot (2020.coling-demos)
Copied to clipboard
| Challenge: | Using conversation with a chatbot, we create annotating and creating datasets through conversation with an open-source platform called Annobot. |
| Approach: | They propose an open-source platform for annotating and creating datasets through conversation with a chatbot. |
| Outcome: | The proposed platform has a wide range of applications including data labelling for binary, multi-class/label classification tasks, preparing data for regression problems and creating sets for issues such as machine translation, question answering or text summarization. |
BotEval: Facilitating Interactive Human Evaluation (2024.acl-demos)
Copied to clipboard
| Challenge: | Using language models to perform complex interactive tasks is becoming more common with the rapid progress in natural language processing (NLP) models. |
| Approach: | They develop an evaluation toolkit that enables human-bot interactions as part of the evaluation process. |
| Outcome: | The evaluation toolkit enables human-bot interactions as part of the evaluation process, rather than making judgements for a static input. |
metaCAT: A Metadata-based Task-oriented Chatbot Annotation Tool (2020.aacl-demo)
Copied to clipboard
| Challenge: | Creating high-quality annotated dialogue corpora necessitates a high level of human engagements. |
| Approach: | They propose to develop an annotation tool specifically for developing task-oriented dialogue data that provides comprehensive metadata annotation coverage to the domain, intent, and span information. |
| Outcome: | The tool provides comprehensive metadata annotation coverage to domain, intent, and span information. |
Addressing Inquiries about History: An Efficient and Practical Framework for Evaluating Open-domain Chatbot Consistency (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to evaluate consistency capacity of open-domain chatbots are costly and low-efficient. |
| Approach: | They propose an efficient framework for evaluating consistency of open-domain chatbots . they use human judges to interact with chatbot, which is costly and low-efficient . |
| Outcome: | The proposed framework can assess the consistency capacity of chatbots and achieve a high ranking correlation with the human evaluation. |
Challenges in Trustworthy Human Evaluation of Chatbots (2025.findings-naacl)
Copied to clipboard
| Challenge: | apathetic or adversarial annotators can corrupt the reliability of open leaderboard rankings . human annotation is widely accepted as the gold standard for open-ended text generation tasks . |
| Approach: | They show that bad annotations can corrupt the reliability of open leaderboard rankings . they argue that human annotation is widely accepted as the gold standard . |
| Outcome: | The proposed algorithm can corrupt the reliability of open leaderboard rankings by up to 5 places. |
TexPrax: A Messaging Application for Ethical, Real-time Data Collection and Annotation (2022.aacl-demo)
Copied to clipboard
Lorenz Stangier, Ji-Ung Lee, Yuxi Wang, Marvin Müller, Nicholas Frick, Joachim Metternich, Iryna Gurevych
| Challenge: | TexPrax is a messaging system to collect and annotate task-oriented dialog data . informal communication channels such as instant messengers are increasingly being used at work . |
| Approach: | They propose a messaging system that collects and annotates task-oriented dialog data from employees via chatbots. |
| Outcome: | The proposed system collects and annotates tasks-oriented dialog data from german factory workers and provides lightweight annotations. |