metaCAT: A Metadata-based Task-oriented Chatbot Annotation Tool (2020.aacl-demo)
Copied to clipboard
| Challenge: | Creating high-quality annotated dialogue corpora necessitates a high level of human engagements. |
| Approach: | They propose to develop an annotation tool specifically for developing task-oriented dialogue data that provides comprehensive metadata annotation coverage to the domain, intent, and span information. |
| Outcome: | The tool provides comprehensive metadata annotation coverage to domain, intent, and span information. |
Similar Papers
EZCAT: an Easy Conversation Annotation Tool (2022.lrec-1)
Copied to clipboard
| Challenge: | EZCAT is an annotation tool for textual conversations, but it is not customizable. |
| Approach: | They propose an easy-to-use interface to annotate conversations in a configurable schema . they use it to annnotate private chats and chats, and they use the schema to test it . |
| Outcome: | The proposed interface allows users to control data and annotate conversations in two levels . it eliminates the need for a server and accounts management, and allows users access to data . |
ChatHF: Collecting Rich Human Feedback from Real-time Conversations (2024.emnlp-demo)
Copied to clipboard
| Challenge: | We present an interactive framework for chatbot evaluation that integrates configurable annotation within a chat interface. |
| Approach: | They propose an interactive framework for chatbot evaluation that integrates configurable annotation within a chat interface. |
| Outcome: | The proposed framework supports fine-grained error detection and human evaluation at the same time. |
A Unifying View On Task-oriented Dialogue Annotation (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent research attention in task-oriented dialogue systems focuses on end-to-end neural models. |
| Approach: | They present a dataset that combines annotated corpora from four domains to provide a unified ontology and annotation schema for task-oriented dialogues. |
| Outcome: | The proposed dataset improves language, information content and performance in dialogues with two recent models. |
Annobot: Platform for Annotating and Creating Datasets through Conversation with a Chatbot (2020.coling-demos)
Copied to clipboard
| Challenge: | Using conversation with a chatbot, we create annotating and creating datasets through conversation with an open-source platform called Annobot. |
| Approach: | They propose an open-source platform for annotating and creating datasets through conversation with a chatbot. |
| Outcome: | The proposed platform has a wide range of applications including data labelling for binary, multi-class/label classification tasks, preparing data for regression problems and creating sets for issues such as machine translation, question answering or text summarization. |
FITAnnotator: A Flexible and Intelligent Text Annotation System (2021.naacl-demos)
Copied to clipboard
| Challenge: | In this paper, we introduce FITAnnotator, a generic web-based tool for efficient text annotation. |
| Approach: | They propose a generic web-based tool for efficient text annotation. |
| Outcome: | The proposed tool is based on a fully modular architecture and provides three kinds of interfaces to annotate instances, evaluate annotation quality and manage the annotation task for annotators, reviewers and managers. |
MultiCAT: Multimodal Communication Annotations for Teams (2025.findings-naacl)
Copied to clipboard
Adarsh Pyarelal, John M Culnan, Ayesha Qamar, Meghavarshini Krishnaswamy, Yuwei Wang, Cheonkam Jeong, Chen Chen, Md Messal Monem Miah, Shahriar Hormozi, Jonathan Tong, Ruihong Huang
| Challenge: | Recent flagship models from OpenAI and Google are only capable of 1-on-1 interactions with humans, limiting the potential for integration into human-machine teams of the future. |
| Approach: | They propose a dataset that allows team members to make multiple types of predictions on the same dataset. |
| Outcome: | The proposed dataset builds upon data from teams working collaboratively to save victims in a simulated search and rescue mission. |
LIDA: Lightweight Interactive Dialogue Annotator (D19-3)
Copied to clipboard
| Challenge: | Dialogue systems are dependent on the quality of the data used to train them. |
| Approach: | They propose to develop an annotation tool specifically for conversation data that handles the entire dialogue annotation pipeline from raw text to structured conversation data. |
| Outcome: | The proposed tool handles the entire dialogue annotation pipeline from raw text to structured conversation data and has a dedicated interface to resolve inter-annotator disagreements. |
TALEN: Tool for Annotation of Low-resource ENtities (P18-4)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a task that requires a large amount of training data and annotators who do not speak the language are hard or impossible to find. |
| Approach: | They propose a web-based interface for named entity annotation in low-resource settings . TALEN includes in-place lexicon integration, TF-IDF token statistics, Internet search, and entity propagation . |
| Outcome: | The proposed interface performs better than a popular annotation tool and is more accurate and recall-rich than the current one. |
LexiClean: An annotation tool for rapid multi-task lexical normalisation (2021.emnlp-demo)
Copied to clipboard
| Challenge: | Lexical normalisation is the task of identifying and normalising non-canonical tokens (e.g. erroneous spelling, acronyms, etc.) in noisy, non-standard, corpora. |
| Approach: | They propose to use LexiClean to annotate multiple tasks in noisy corpora using in situ token modification and annotation that can be rapidly applied corpus wide. |
| Outcome: | The proposed tool can be rapidly applied corpus wide and can identify and normalise noisy, non-standard, and domain specific corpora. |
Dragonfly: Advances in Non-Speaker Annotation for Low Resource Languages (2020.lrec-1)
Copied to clipboard
| Challenge: | Using semantic and contextual information, non-speakers of a language familiar with the Latin script can produce high quality named entity annotations to support construction of . name tagger. |
| Approach: | They propose a procedure for annotating low resource languages using Dragonfly that others can use. |
| Outcome: | The proposed procedure improves the performance of NER models on native speaker and non-speaker annotations in low resource languages. |