| Challenge: | Developing dialogue systems such as Apple Siri and Google Now requires high quality training data but data collection with crowdsourcing is largely an open question. |
| Approach: | They propose to use crowdsourcing to collect data for a user intent classification task in a dialogue system. |
| Outcome: | The proposed method improves the quality of the collected data and the model performance on real user queries. |
Similar Papers
Dialogue Scenario Collection of Persuasive Dialogue with Emotional Expressions via Crowdsourcing (L18-1)
Copied to clipboard
| Challenge: | Existing methods for data collection and annotation are costly and prevent launching new dialogue systems. |
| Approach: | They asked crowd workers to create persuasive dialogue systems using emotional expressions . they annotated emotional states and users' acceptance for system persuasion . |
| Outcome: | The proposed system has sufficient agreement even without training, the researchers found . the experiment showed that the collected data are comparable to real-world dialogue recording methods . |
DialCrowd 2.0: A Quality-Focused Dialog System Crowdsourcing Toolkit (2022.lrec-1)
Copied to clipboard
| Challenge: | DialCrowd 2.0 helps requesters obtain higher quality data from human intelligence tasks. |
| Approach: | They propose to use DialCrowd 2.0 to help requesters obtain higher quality data . they aim to improve the way requesters present tasks and facilitate effective communication with workers. |
| Outcome: | The proposed toolkit enables requesters to obtain higher quality data by presenting tasks more clearly and facilitating effective communication with workers. |
PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations (2023.findings-acl)
Copied to clipboard
| Challenge: | Mars? - PragmatiCQA |
| Approach: | Mars? - The Paper . |
| Outcome: | The proposed dataset features 6873 QA pairs that explores pragmatic reasoning in conversations over a diverse set of topics. |
Data Collection and End-to-End Learning for Conversational AI (D19-2)
Copied to clipboard
| Challenge: | tutorial aims to familiarise research community with recent advances in statistical dialogue systems . focus of tutorial is on learning end-to-end from data and their relation to more common modular systems. |
| Approach: | This tutorial aims to familiarise the research community with the latest advances in statistical dialogue systems . the focus of the tutorial is on recently introduced end-to-end learning for dialogue systems and their relation to more common modular systems. |
| Outcome: | This tutorial aims to familiarise the research community with the recent advances in statistical dialogue systems for open-domain and task-based dialogue paradigms. |
Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset (D19-1)
Copied to clipboard
Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, Andy Cedilnik
| Challenge: | a lack of high quality conversational data is limiting progress in dialog systems . we present a dataset of 13,215 task-based dialogs . |
| Approach: | They propose a task-based dialog dataset which includes 13,215 task-related dialogs . they use a two-person, spoken "Wizard of Oz" approach and a "self-dialog" approach . |
| Outcome: | The taskmaster-1 dataset contains 13,215 task-based dialogs comprising six domains. |
Deep Learning for Dialogue Systems (C18-3)
Copied to clipboard
| Challenge: | Using deep learning to build robust and scalable spoken dialogue systems is still a challenging task. |
| Approach: | tutorial focuses on an overview of dialogue system development . goal-oriented spoken dialogue systems are most prominent component in virtual personal assistants . |
| Outcome: | This tutorial focuses on an overview of dialogue system development while summarizing the challenges. |
Crowdsourcing Beyond Annotation: Case Studies in Benchmark Data Collection (2021.emnlp-tutorials)
Copied to clipboard
| Challenge: | Developing a theory of crowdsourcing for practical language problems remains an open challenge . |
| Approach: | This tutorial exposes NLP researchers to data collection crowdsourcing methods and principles through case studies. |
| Outcome: | This tutorial exposes NLP researchers to various data collection crowdsourcing methods and practices through case studies. |
Going beyond research datasets: Novel intent discovery in the industry setting (2023.findings-eacl)
Copied to clipboard
| Challenge: | Novel intent discovery automates grouping of similar messages to identify previously unknown intents. |
| Approach: | They propose to use question-only data to improve the intent discovery pipeline . they propose to utilize conversational structure of real-life datasets for clustering . |
| Outcome: | The proposed method gives 33pp performance boost over state-of-the-art model for question only . it also gives 13pp performance increase over the naive baseline model . |
ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness (2023.emnlp-main)
Copied to clipboard
| Challenge: | generative large language models (LLMs) are replacing human workers for some tasks . crowdsourcing has several downsides: 1) the workforce is costly, 2) output quality is difficult to achieve, and 3) there are overheads related to the design and organization of the process. |
| Approach: | They investigate whether ChatGPT-created paraphrases are more diverse and robust . they use a crowdsourcing tool to collect training or validation examples . |
| Outcome: | The proposed models are more diverse and robust than the existing models. |
More Diverse Dialogue Datasets via Diversity-Informed Data Collection (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to generate conversational dialogue produce uninteresting, predictable responses. |
| Approach: | They propose a method to collect and determine more diverse data from conversational participants . they use dynamically computed corpus-level statistics to determine which conversational participant to collect data from . |
| Outcome: | The proposed method produces significantly more diverse data than baseline methods and better results on emotion classification and dialogue generation tasks. |