| Challenge: | Recent research attention in task-oriented dialogue systems focuses on end-to-end neural models. |
| Approach: | They present a dataset that combines annotated corpora from four domains to provide a unified ontology and annotation schema for task-oriented dialogues. |
| Outcome: | The proposed dataset improves language, information content and performance in dialogues with two recent models. |
Similar Papers
Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation (2023.tacl-1)
Copied to clipboard
| Challenge: | Multilingual task-oriented dialogue (ToD) datasets suffer from severe limitations, such as being small in scale and lacking naturalness and cultural specificity in the target language. |
| Approach: | They propose a novel outline-based annotation process where domain-specific abstract schemata of dialogue are mapped into natural language outlines. |
| Outcome: | The proposed approach improves understanding, dialogue state tracking, and end-to-end dialogue evaluation in Arabic, Indonesian, Russian, and Kiswahili. |
MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling (D18-1)
Copied to clipboard
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, Milica Gašić
| Challenge: | a dataset of 10k human-human written conversations is one order of magnitude larger than previous annotated task-oriented corpora. |
| Approach: | They propose to collect 10k human-human written conversations from a crowd-sourced dataset using crowd-sourcing. |
| Outcome: | The proposed dataset is one order of magnitude larger than previous annotated task-oriented corpora and shows the usability of the data and sets a baseline for future studies. |
RiSAWOZ: A Large-Scale Multi-Domain Wizard-of-Oz Dataset with Rich Semantic Annotations for Task-Oriented Dialogue Modeling (2020.emnlp-main)
Copied to clipboard
| Challenge: | RiSAWOZ contains 11.2K human-to-human (H2H) multi-turn semantically annotated dialogues spanning over 12 domains . despite of substantial progress made, there are challenges in creating challenging datasets in terms of size, multiple domains, semantic annotations and complexity. |
| Approach: | They propose a large-scale multi-domain Chinese Wizard-of-Oz dataset with rich semantic annotations that captures discourse phenomena for task-oriented dialogue modeling. |
| Outcome: | The proposed dataset contains 11.2K human-to-human (H2H) multi-turn semantically annotated dialogues with more than 150K utterances spanning over 12 domains. |
MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Baselines (2020.lrec-1)
Copied to clipboard
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, Dilek Hakkani-Tur
| Challenge: | MultiWOZ 2.0 has substantial noise in dialogue state annotations and dialogue utterances . follow-up work has augmented the original dataset with user dialogue acts . |
| Approach: | They propose to reannotate dialogue state and utterances based on original dataset . they then compare their results to other datasets to improve their models . |
| Outcome: | The proposed dataset improves on the noise in the dialogue state annotations and dialogue utterances. |
Unsupervised End-to-End Task-Oriented Dialogue with LLMs: The Power of the Noisy Channel (2024.emnlp-main)
Copied to clipboard
| Challenge: | a task-oriented dialogue system requires turn-level annotations for interacting with their APIs. |
| Approach: | They propose an unsupervised approach that infers turn-level annotations as latent variables using a noisy channel model to build an end-to-end dialogue agent. |
| Outcome: | The proposed method doubles the success rate of a strong GPT-3.5 benchmark. |
JMultiWOZ: A Large-Scale Japanese Multi-Domain Task-Oriented Dialogue Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing datasets for task-oriented dialogue systems in English are limited compared to Japanese. |
| Approach: | They evaluated the dialogue state tracking and response generation capabilities of Japanese language datasets using multi-domain task-oriented dialogues. |
| Outcome: | The proposed dataset provides a benchmark that is on par with MultiWOZ2.2 and the latest large language model (LLM)-based methods. |
DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI (2024.findings-eacl)
Copied to clipboard
Jianguo Zhang, Kun Qian, Zhiwei Liu, Shelby Heinecke, Rui Meng, Ye Liu, Zhou Yu, Huan Wang, Silvio Savarese, Caiming Xiong
| Challenge: | DialogStudio is the largest and most diverse collection of dialogue datasets . existing datasets lack diversity and comprehensiveness, authors say . |
| Approach: | They introduce DialogStudio: the largest and most diverse collection of dialogue datasets . DialogStuio aggregates more than 80 diverse dialogue dataset . |
| Outcome: | a new dataset is created to improve the quality and diversity of dialogue datasets . DialogStudio is the largest and most diverse collection of dialogue data . |
EmoWOZ: A Large-Scale Corpus and Labelling Scheme for Emotion Recognition in Task-Oriented Dialogue Systems (2022.lrec-1)
Copied to clipboard
Shutong Feng, Nurul Lubis, Christian Geishauser, Hsien-chin Lin, Michael Heck, Carel van Niekerk, Milica Gasic
| Challenge: | Existing emotion-annotated task-oriented corpora are limited in size, label richness, and public availability, creating a bottleneck for downstream tasks. |
| Approach: | They propose a large-scale manually emotion-annotated corpus of task-oriented dialogues based on a multi-domain task-orientated dataset. |
| Outcome: | The proposed method is based on a task-oriented dialogue dataset with 11K dialogues and 83K emotion annotations of user utterances. |
Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing goal-oriented dialogue datasets focus on identifying slots and values, but in reality, customer service agents follow multi-step procedures derived from explicit company policies. |
| Approach: | They propose to use a fully-labeled dataset to study customer service dialogue systems in real-world scenarios. |
| Outcome: | The proposed dataset outperforms existing models but still lacks 50.8% absolute accuracy to reach human-level performance on the dataset. |
Alexa Conversations: An Extensible Data-driven Approach for Building Task-oriented Dialogue Systems (2021.naacl-demos)
Copied to clipboard
Anish Acharya, Suranjit Adhikari, Sanchit Agarwal, Vincent Auvray, Nehal Belgamwar, Arijit Biswas, Shubhra Chandra, Tagyoung Chung, Maryam Fazel-Zarandi, Raefer Gabriel, Shuyang Gao, Rahul Goel, Dilek Hakkani-Tur, Jan Jezabek, Abhay Jha, Jiun-Yu Kao, Prakash Krishnan, Peter Ku, Anuj Goyal, Chien-Wei Lin, Qing Liu, Arindam Mandal, Angeliki Metallinou, Vishal Naik, Yi Pan, Shachi Paul, Vittorio Perera, Abhishek Sethi, Minmin Shen, Nikko Strom, Eddie Wang
| Challenge: | Traditional goal-oriented dialogue systems require annotations which are hard to obtain for every new domain, limiting scalability. |
| Approach: | They propose a data-driven approach to building goal-oriented dialogue systems . they use a seed dialogue simulator to generate annotated conversations instead of collecting annotations . |
| Outcome: | The proposed system improves turn-level action signature prediction accuracy by 50% . the system is scalable, extensible and data efficient . |