Papers with Crowdsourcing
Crowdsourcing Beyond Annotation: Case Studies in Benchmark Data Collection (2021.emnlp-tutorials)
Copied to clipboard
| Challenge: | Developing a theory of crowdsourcing for practical language problems remains an open challenge . |
| Approach: | This tutorial exposes NLP researchers to data collection crowdsourcing methods and principles through case studies. |
| Outcome: | This tutorial exposes NLP researchers to various data collection crowdsourcing methods and practices through case studies. |
Dependency Tree Annotation with Mechanical Turk (D19-59)
Copied to clipboard
| Challenge: | a recent study shows that crowdsourcing is often used to obtain linguistic annotations but is rarely used for parsing. |
| Approach: | They propose to use Mechanical Turk to crowdsource parse trees using an interactive graphical dependency tree editor. |
| Outcome: | The proposed method is the first published use of Mechanical Turk to crowdsource parse trees . the authors find that the workers achieve high levels of accuracy on 72% of the sentences . |
Data Collection for Dialogue System: A Startup Perspective (N18-3)
Copied to clipboard
| Challenge: | Developing dialogue systems such as Apple Siri and Google Now requires high quality training data but data collection with crowdsourcing is largely an open question. |
| Approach: | They propose to use crowdsourcing to collect data for a user intent classification task in a dialogue system. |
| Outcome: | The proposed method improves the quality of the collected data and the model performance on real user queries. |
A Study of Incorrect Paraphrases in Crowdsourced User Utterances (N19-1)
Copied to clipboard
| Challenge: | Developing bots requires high quality training samples, especially for unqualified crowd workers. |
| Approach: | They propose an annotated dataset for detecting quality issues in crowdsourced paraphrasing . they propose to use existing tools and services to provide baselines for identifying issues . |
| Outcome: | The proposed dataset provides a baseline for detecting unqualified paraphrases. |
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021.acl-long)
Copied to clipboard
| Challenge: | Despite the importance of datasets for natural language understanding, there has been little attention on crowdsourcing methods for collecting datasets. |
| Approach: | They compare the effectiveness of crowdsourcing methods for boosting NLU example difficulty with training crowdworkers instead of expert judgments. |
| Outcome: | The proposed method is ineffective for boosting NLU example difficulty, but it is not effective for training crowdworkers and qualifying workers based on expert judgments. |
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets (D19-1)
Copied to clipboard
| Challenge: | Having only a few workers generate the majority of dataset examples raises concerns about data diversity . |
| Approach: | They perform a series of experiments to investigate annotator biases in recent NLU datasets . they find that models are able to recognize the most productive annotators . |
| Outcome: | The results show that models can recognize the most productive annotators and do not generalize well to examples from annotator that did not contribute to the training set. |
Exploring the Cost-Effectiveness of Perspective Taking in Crowdsourcing Subjective Assessment: A Case Study of Toxicity Detection (2025.naacl-long)
Copied to clipboard
| Challenge: | toxicity evaluation tasks require annotations to accurately reflect opinions of subgroups . toxicity tasks require annotators to take the opinions of a subgroup simultaneously . |
| Approach: | They propose to use perspective taking to obtain opinions from subgroups . they propose to prompt annotators to take perspectives of contrasting subgroup simultaneously . |
| Outcome: | The proposed approach can be cost-effective and improve quality under limited budget. |
Toward Annotator Group Bias in Crowdsourcing (2022.acl-long)
Copied to clipboard
Haochen Liu, Joseph Thekinen, Sinem Mollaoglu, Da Tang, Ji Yang, Youlong Cheng, Hui Liu, Jiliang Tang
| Challenge: | Annotator group bias is a common problem in crowdsourcing, but is often overlooked . |
| Approach: | They propose a probabilistic framework to capture annotator group bias using an extended Expectation Maximization algorithm. |
| Outcome: | The proposed model can model annotator group bias over competitive datasets and demonstrate that it is effective over multiple datasets. |
Identifying Chinese Opinion Expressions with Extremely-Noisy Crowdsourcing Annotations (2022.acl-long)
Copied to clipboard
| Challenge: | Recent works of opinion expression identification (OEI) rely heavily on the quality and scale of the manually-constructed training corpus. |
| Approach: | They propose to use crowdsourcing annotations to build a large-scale but quality-unguaranteed corpus for opinion expression identification in Chinese. |
| Outcome: | The proposed model can be trained with a synthetic expert and is highly consistent with the training and testing phase. |
A Probabilistic Annotation Model for Crowdsourcing Coreference (D18-1)
Copied to clipboard
| Challenge: | Existing methods to generate annotated corpora for coreference are expensive and limited. |
| Approach: | They propose a model of annotation for aggregating crowdsourced anaphoric annotations. |
| Outcome: | The proposed model can extract from crowdsourced annotations coreference chains comparable to those obtained with expert annotation. |
Crowdsourcing Learning as Domain Adaptation: A Case Study on Named Entity Recognition (2021.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that crowdsourced annotations are highly effective under supervised conditions. |
| Approach: | They propose an annotator-aware representation learning model that is inspired by domain adaptation methods which attempt to capture effective domain-alike features. |
| Outcome: | The proposed model is highly effective on a benchmark dataset and achieves state-of-the-art performance with only a very small scale of expert annotations. |
A Neural Model for Aggregating Coreference Annotation in Crowdsourcing (2020.coling-main)
Copied to clipboard
| Challenge: | Existing studies of natural language labelling tasks have shown that crowd-sourced labels can be noisy. |
| Approach: | They split the aggregation into mention classification and coreference chain inference tasks to predict the correct labels. |
| Outcome: | The proposed model predicts the class of each mention using an autoencoder while taking into account the mention’s annotation complexity and annotators’ reliability at different levels. |
Mining Crowdsourcing Problems from Discussion Forums of Workers (2020.coling-main)
Copied to clipboard
| Challenge: | Among the most widely used platforms are Upwork, Appen, and above all Amazon Mechanical Turk (MTurk) which host annotation tasks and collect huge sets of annotated data from workers. |
| Approach: | They propose to use topic modeling to analyze workers' complaints from a new English corpus of workers’ forum discussions to identify problems in task design, task operation, and task evaluation that workers face with requesters in crowdsourcing processes. |
| Outcome: | The findings form the basis for future research on how to improve crowdsourcing processes. |