Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, Andy Cedilnik
| Challenge: | a lack of high quality conversational data is limiting progress in dialog systems . we present a dataset of 13,215 task-based dialogs . |
| Approach: | They propose a task-based dialog dataset which includes 13,215 task-related dialogs . they use a two-person, spoken "Wizard of Oz" approach and a "self-dialog" approach . |
| Outcome: | The taskmaster-1 dataset contains 13,215 task-based dialogs comprising six domains. |
Similar Papers
MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling (D18-1)
Copied to clipboard
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, Milica Gašić
| Challenge: | a dataset of 10k human-human written conversations is one order of magnitude larger than previous annotated task-oriented corpora. |
| Approach: | They propose to collect 10k human-human written conversations from a crowd-sourced dataset using crowd-sourcing. |
| Outcome: | The proposed dataset is one order of magnitude larger than previous annotated task-oriented corpora and shows the usability of the data and sets a baseline for future studies. |
DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI (2024.findings-eacl)
Copied to clipboard
Jianguo Zhang, Kun Qian, Zhiwei Liu, Shelby Heinecke, Rui Meng, Ye Liu, Zhou Yu, Huan Wang, Silvio Savarese, Caiming Xiong
| Challenge: | DialogStudio is the largest and most diverse collection of dialogue datasets . existing datasets lack diversity and comprehensiveness, authors say . |
| Approach: | They introduce DialogStudio: the largest and most diverse collection of dialogue datasets . DialogStuio aggregates more than 80 diverse dialogue dataset . |
| Outcome: | a new dataset is created to improve the quality and diversity of dialogue datasets . DialogStudio is the largest and most diverse collection of dialogue data . |
Multi 3 WOZ: A Multilingual, Multi-Domain, Multi-Parallel Dataset for Training and Evaluating Culturally Adapted Task-Oriented Dialog Systems (2023.tacl-1)
Copied to clipboard
Songbo Hu, Han Zhou, Mete Hergul, Milan Gritta, Guchun Zhang, Ignacio Iacobacci, Ivan Vulić, Anna Korhonen
| Challenge: | Task-oriented dialog (TOD) is one of the central objectives, hallmarks, and applications of machine intelligence. |
| Approach: | They propose a multilingual, multi-domain, multiparallele ToD dataset that offers culturally adapted dialogs in 4 languages for training and evaluation of multilingual and cross-lingual systems. |
| Outcome: | The proposed dataset is large-scale and culturally adapted to enable training and evaluation of multilingual and cross-lingual ToD systems. |
DialCrowd 2.0: A Quality-Focused Dialog System Crowdsourcing Toolkit (2022.lrec-1)
Copied to clipboard
| Challenge: | DialCrowd 2.0 helps requesters obtain higher quality data from human intelligence tasks. |
| Approach: | They propose to use DialCrowd 2.0 to help requesters obtain higher quality data . they aim to improve the way requesters present tasks and facilitate effective communication with workers. |
| Outcome: | The proposed toolkit enables requesters to obtain higher quality data by presenting tasks more clearly and facilitating effective communication with workers. |
PRESTO: A Multilingual Dataset for Parsing Realistic Task-Oriented Dialogs (2023.emnlp-main)
Copied to clipboard
Rahul Goel, Waleed Ammar, Aditya Gupta, Siddharth Vashishtha, Motoki Sano, Faiz Surani, Max Chang, HyunJeong Choe, David Greene, Chuan He, Rattima Nitisaroj, Anna Trukhina, Shachi Paul, Pararth Shah, Rushin Shah, Zhou Yu
| Challenge: | PRESTO dataset contains 550K contextual multilingual conversations between humans and virtual assistants. |
| Approach: | They propose to use a dataset of 550K contextual multilingual conversations between humans and virtual assistants to study some of the more challenging aspects of parsing realistic conversations. |
| Outcome: | The dataset contains 550K contextual conversations between humans and virtual assistants. |
CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset (2020.tacl-1)
Copied to clipboard
| Challenge: | Despite the significant contributions to the community, there is still a gap between existing dialogue corpora and real-life human dialogue data. |
| Approach: | They propose to develop Chinese cross-domain wizard-of-oz task-oriented dataset CrossWOZ with rich annotations of dialogue states and dialogue acts on both user and system sides. |
| Outcome: | The proposed dataset contains 6K dialogue sessions and 102K utterances for 5 domains, including hotel, restaurant, attraction, metro, and taxi. |
Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing goal-oriented dialogue datasets focus on identifying slots and values, but in reality, customer service agents follow multi-step procedures derived from explicit company policies. |
| Approach: | They propose to use a fully-labeled dataset to study customer service dialogue systems in real-world scenarios. |
| Outcome: | The proposed dataset outperforms existing models but still lacks 50.8% absolute accuracy to reach human-level performance on the dataset. |
SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing task-oriented dialog datasets do not situate the dialog in the user’s multimodal context. |
| Approach: | They propose to use a dataset to study multimodal task-oriented dialogs in the shopping domain to situate them in the user’s multimodal context. |
| Outcome: | The proposed dataset includes 11K task-oriented user->assistant dialogs (117K utterances) in the shopping domain, grounded in immersive and photo-realistic scenes. |
A Unifying View On Task-oriented Dialogue Annotation (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent research attention in task-oriented dialogue systems focuses on end-to-end neural models. |
| Approach: | They present a dataset that combines annotated corpora from four domains to provide a unified ontology and annotation schema for task-oriented dialogues. |
| Outcome: | The proposed dataset improves language, information content and performance in dialogues with two recent models. |
doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset (2020.emnlp-main)
Copied to clipboard
| Challenge: | doc2dial dataset is a goal-oriented document-grounded dialogue model . it is based on how the authors compose documents for guiding end users . |
| Approach: | They propose a dataset of goal-oriented dialogues grounded in documents . they use annotated conversations with an average of 14 turns to generate conversational utterances . |
| Outcome: | The proposed dataset includes over 4500 annotated conversations with an average of 14 turns grounded in over 450 documents from four domains. |