DialCrowd 2.0: A Quality-Focused Dialog System Crowdsourcing Toolkit (2022.lrec-1)
Copied to clipboard
| Challenge: | DialCrowd 2.0 helps requesters obtain higher quality data from human intelligence tasks. |
| Approach: | They propose to use DialCrowd 2.0 to help requesters obtain higher quality data . they aim to improve the way requesters present tasks and facilitate effective communication with workers. |
| Outcome: | The proposed toolkit enables requesters to obtain higher quality data by presenting tasks more clearly and facilitating effective communication with workers. |
Similar Papers
Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset (D19-1)
Copied to clipboard
Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, Andy Cedilnik
| Challenge: | a lack of high quality conversational data is limiting progress in dialog systems . we present a dataset of 13,215 task-based dialogs . |
| Approach: | They propose a task-based dialog dataset which includes 13,215 task-related dialogs . they use a two-person, spoken "Wizard of Oz" approach and a "self-dialog" approach . |
| Outcome: | The taskmaster-1 dataset contains 13,215 task-based dialogs comprising six domains. |
Data Collection for Dialogue System: A Startup Perspective (N18-3)
Copied to clipboard
| Challenge: | Developing dialogue systems such as Apple Siri and Google Now requires high quality training data but data collection with crowdsourcing is largely an open question. |
| Approach: | They propose to use crowdsourcing to collect data for a user intent classification task in a dialogue system. |
| Outcome: | The proposed method improves the quality of the collected data and the model performance on real user queries. |
Are the Tools up to the Task? an Evaluation of Commercial Dialog Tools in Developing Conversational Enterprise-grade Dialog Systems (N19-2)
Copied to clipboard
| Challenge: | Existing toolsets are incomplete in meeting the goal of building effective dialog systems, authors say . |
| Approach: | They compare dialog tools available from a number of companies to determine their strengths and weaknesses . they provide quantitative and qualitative results in three main areas: natural language understanding, dialog, and text generation . |
| Outcome: | The toolsets are incomplete, but they are compared to other tools to determine their strengths and weaknesses. |
A Study of Incorrect Paraphrases in Crowdsourced User Utterances (N19-1)
Copied to clipboard
| Challenge: | Developing bots requires high quality training samples, especially for unqualified crowd workers. |
| Approach: | They propose an annotated dataset for detecting quality issues in crowdsourced paraphrasing . they propose to use existing tools and services to provide baselines for identifying issues . |
| Outcome: | The proposed dataset provides a baseline for detecting unqualified paraphrases. |
q2d: Turning Questions into Dialogs to Teach Models How to Search (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent dialog generation models use external search APIs to generate grounded responses. |
| Approach: | They propose an automatic data generation pipeline that generates dialogs from questions . they use a large language model to create conversational versions of question answering datasets . |
| Outcome: | The proposed method improves query generation models on a QReCC dataset. |
TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems (2021.acl-long)
Copied to clipboard
| Challenge: | TicketTalk dataset with 23,789 annotated dialogs is a data-driven, end-to-end approach to transaction-based dialog systems that performs at near-human levels in terms of verbal response quality and factual grounding accuracy. |
| Approach: | They propose a data-driven, end-to-end approach to transaction-based dialog systems that performs at near-human levels in terms of verbal response quality and factual grounding accuracy. |
| Outcome: | The proposed model generates verbal responses and API call predictions on a movie ticketing dialog dataset with 23,789 annotated conversations. |
Gated Mechanism Enhanced Multi-Task Learning for Dialog Routing (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for dialog routing are mostly heuristic and cannot achieve high-quality performance. |
| Approach: | They propose a multi-task learning framework with a dialog encoder and two tailored gated mechanism modules to solve this problem. |
| Outcome: | The proposed model can play the role of hierarchical information filtering and is non-invasive to existing dialog systems. |
What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing metrics for dialog evaluation are trained on human annotations, which is cumbersome to collect. |
| Approach: | They propose to use user sentiment and other information as proxy to measure the quality of previous dialogs. |
| Outcome: | The proposed model is comparable to models trained on human annotated data. |
What we need to learn if we want to do and not just talk (N18-3)
Copied to clipboard
| Challenge: | Existing methods for task-oriented dialogs require fluent natural language responses and correct external actions . but they perform poorly in real world dialog tasks, a new study shows . |
| Approach: | They propose a hybrid model where nearest neighbor is used to generate fluent responses and Seq2Seq type models ensure dialogue coherency and generate accurate external actions. |
| Outcome: | The proposed model achieves a 78% relative improvement in fluency and 200% improvement in accuracy of external calls. |
Mining Crowdsourcing Problems from Discussion Forums of Workers (2020.coling-main)
Copied to clipboard
| Challenge: | Among the most widely used platforms are Upwork, Appen, and above all Amazon Mechanical Turk (MTurk) which host annotation tasks and collect huge sets of annotated data from workers. |
| Approach: | They propose to use topic modeling to analyze workers' complaints from a new English corpus of workers’ forum discussions to identify problems in task design, task operation, and task evaluation that workers face with requesters in crowdsourcing processes. |
| Outcome: | The findings form the basis for future research on how to improve crowdsourcing processes. |