Annobot: Platform for Annotating and Creating Datasets through Conversation with a Chatbot (2020.coling-demos)
Copied to clipboard
| Challenge: | Using conversation with a chatbot, we create annotating and creating datasets through conversation with an open-source platform called Annobot. |
| Approach: | They propose an open-source platform for annotating and creating datasets through conversation with a chatbot. |
| Outcome: | The proposed platform has a wide range of applications including data labelling for binary, multi-class/label classification tasks, preparing data for regression problems and creating sets for issues such as machine translation, question answering or text summarization. |
Similar Papers
ChatHF: Collecting Rich Human Feedback from Real-time Conversations (2024.emnlp-demo)
Copied to clipboard
| Challenge: | We present an interactive framework for chatbot evaluation that integrates configurable annotation within a chat interface. |
| Approach: | They propose an interactive framework for chatbot evaluation that integrates configurable annotation within a chat interface. |
| Outcome: | The proposed framework supports fine-grained error detection and human evaluation at the same time. |
EZCAT: an Easy Conversation Annotation Tool (2022.lrec-1)
Copied to clipboard
| Challenge: | EZCAT is an annotation tool for textual conversations, but it is not customizable. |
| Approach: | They propose an easy-to-use interface to annotate conversations in a configurable schema . they use it to annnotate private chats and chats, and they use the schema to test it . |
| Outcome: | The proposed interface allows users to control data and annotate conversations in two levels . it eliminates the need for a server and accounts management, and allows users access to data . |
metaCAT: A Metadata-based Task-oriented Chatbot Annotation Tool (2020.aacl-demo)
Copied to clipboard
| Challenge: | Creating high-quality annotated dialogue corpora necessitates a high level of human engagements. |
| Approach: | They propose to develop an annotation tool specifically for developing task-oriented dialogue data that provides comprehensive metadata annotation coverage to the domain, intent, and span information. |
| Outcome: | The tool provides comprehensive metadata annotation coverage to domain, intent, and span information. |
Recipes for Building an Open-Domain Chatbot (2021.eacl-main)
Copied to clipboard
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, Jason Weston
| Challenge: | Existing work shows that scaling models in the number of parameters and the size of the data they are trained on gives improved results, but other factors are important. |
| Approach: | They propose to build open-domain chatbots that can be scaled to improve their performance . they use a blend of cognitive and cognitive skills to build a model that combines these skills . |
| Outcome: | The proposed models outperform existing approaches in multi-turn dialogue on engagingness and humanness measurements. |
Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems (2020.emnlp-main)
Copied to clipboard
Jan Deriu, Don Tuggener, Pius von Däniken, Jon Ander Campos, Alvaro Rodrigo, Thiziri Belkacem, Aitor Soroa, Eneko Agirre, Mark Cieliebak
| Challenge: | Lack of time efficient and reliable evalu-ation methods is hampering the development of conversational dialogue systems (chatbots). |
| Approach: | They propose a framework that replaces human-bot conversations with conversations between bots and an annotation tool that ranks chatbots based on their ability to mimic human behaviour. |
| Outcome: | The proposed evaluation framework replaces human-bot conversations with bot conversations and allows for frequent evaluations of chatbots during their evaluation cycle. |
ChatEval: A Tool for Chatbot Evaluation (N19-4)
Copied to clipboard
| Challenge: | open-domain dialog systems are difficult to evaluate due to lack of standardization and standardization in evaluation procedures. |
| Approach: | They propose a framework for human evaluation of chatbots that augments existing tools . researchers can submit their trained models to the ChatEval web interface . reproducibility and model assessment for opendomain dialog systems is challenging . |
| Outcome: | The proposed framework provides a web-based hub for researchers to compare their models with baselines and prior work. |
Label efficient semi-supervised conversational intent classification (2023.acl-industry)
Copied to clipboard
| Challenge: | A conversational chatbot can answer pre-purchase questions and post-purchase queries to provide a seamless shopping experience. |
| Approach: | They propose a semi-supervised learning approach for label-efficient intent classification using a small labeled corpus and large unlabeled query data to train a transformer model. |
| Outcome: | The proposed approach significantly improves over the baseline, even with a limited labeled set. |
The slurk Interaction Server Framework: Better Data for Better Dialog Models (2022.lrec-1)
Copied to clipboard
| Challenge: | slurk is a lightweight dialog data collection and testing tool for crowdsourcing platforms. |
| Approach: | They present a lightweight dialog server that allows to set up dialog data collections and run experiments. |
| Outcome: | The slurk software allows to set up dialog data collections and run experiments with no limitations on the number of participants. |
LIDA: Lightweight Interactive Dialogue Annotator (D19-3)
Copied to clipboard
| Challenge: | Dialogue systems are dependent on the quality of the data used to train them. |
| Approach: | They propose to develop an annotation tool specifically for conversation data that handles the entire dialogue annotation pipeline from raw text to structured conversation data. |
| Outcome: | The proposed tool handles the entire dialogue annotation pipeline from raw text to structured conversation data and has a dedicated interface to resolve inter-annotator disagreements. |
The R-U-A-Robot Dataset: Helping Avoid Chatbot Deception by Detecting User Questions About Human or Non-Human Identity (2021.acl-long)
Copied to clipboard
| Challenge: | We analyze 2,500 phrasings related to the intent of “Are you a robot?” and 2,500 adversarially selected utterances to determine whether systems are non-human. |
| Approach: | They analyze 2,500 phrasings related to the intent of "Are you a robot?" and 2,500 adversarially selected utterances to determine whether systems are non-human. |
| Outcome: | The proposed model and two systems fail to confirm non-human intent, and the proposed model is complex. |