A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Prior attempts to develop crowdsourcing methods have either had low accuracy or required substantial expert annotation. |
| Approach: | They propose a multi-stage crowd workflow that reduces expert involvement without sacrificing accuracy. |
| Outcome: | The proposed method reduces expert effort by 4x, from 56% to 14% of cases. |
Similar Papers
Controlled Crowdsourcing for High-Quality QA-SRL Annotation (2020.acl-main)
Copied to clipboard
Paul Roit, Ayal Klein, Daniela Stepanov, Jonathan Mamou, Julian Michael, Gabriel Stanovsky, Luke Zettlemoyer, Ido Dagan
| Challenge: | Question-answer driven Semantic Role Labeling (QA-SRL) is an open and natural flavour of SRL, potentially attainable from laymen. |
| Approach: | They propose a question-answer driven semantic role labeling approach that uses question-announced questions to label predicate-argument relationships. |
| Outcome: | The proposed method yields high-quality annotation with dramatically higher coverage, enabling future replicable research of natural semantic annotations. |
SRL4ORL: Improving Opinion Role Labeling Using Multi-Task Learning with Semantic Role Labeling (N18-1)
Copied to clipboard
| Challenge: | Recent neural approaches do not outperform the state-of-the-art feature-based models for Opinion Role Labeling (ORL). |
| Approach: | They propose to use multi-task learning to improve Opinion Role Labeling by using a related task which has substantially more data. |
| Outcome: | The proposed model outperforms the state-of-the-art model for Opinion Role Labeling (ORL) with more data. |
Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types (C18-1)
Copied to clipboard
| Challenge: | Annotated corpora are often assigned to internet workers whose judgments are reconciled by crowdsourcing models. |
| Approach: | They propose a framework for learning from rich prior knowledge to combine annotations with different structures. |
| Outcome: | The proposed model compares favorably with previous work and enables active sample selection to reduce annotation effort. |
Semantic Role Labeling Meets Definition Modeling: Using Natural Language to Describe Predicate-Argument Structures (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to Semantic Role Labeling rely on discrete labels to classify predicate senses and their arguments. |
| Approach: | They propose a generalized formulation of Semantic Role Labeling that leverages Definition Modeling to describe predicate-argument structures using natural language definitions instead of discrete labels. |
| Outcome: | The proposed model can describe predicate-argument structures using natural language definitions instead of discrete labels. |
Label Definitions Improve Semantic Role Labeling (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing work on semantic role labeling treats symbolic labels as symbolic . labeled data is costly and often lacking in many tasks, domains, and languages. |
| Approach: | They propose to retrieve and leverage semantic role labels from annotation guidelines . argument classification is at the core of Semantic Role Labeling . |
| Outcome: | The proposed model achieves state-of-the-art on a CoNLL09 dataset injected with label definitions given the predicate senses. |
Crowd-sourcing annotation of complex NLU tasks: A case study of argumentative content annotation (D19-59)
Copied to clipboard
| Challenge: | Recent advances in machine reading and listening comprehension involve the annotation of long texts. |
| Approach: | They propose a way to perform a sentence-by-sentence annotation task with crowd annotators. |
| Outcome: | The proposed approach can be used to identify claims in a debate speech. |
Exploring Non-Verbal Predicates in Semantic Role Labeling: Challenges and Opportunities (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing systems for SRL are incapable of transferring knowledge across different predicate types. |
| Approach: | They propose a new PropBank dataset which boasts wide coverage of multiple predicate types and a manually-annotated challenge set which gives equal importance to verbal, nominal, and adjectival predicates. |
| Outcome: | The proposed dataset shows that standard benchmarks do not provide an accurate picture of the current situation in SRL and that state-of-the-art systems are still incapable of transferring knowledge across different predicate types. |
Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution (2020.coling-main)
Copied to clipboard
| Challenge: | a scalable method for estimating the noisiness of labels produced by crowdsourcing annotation tasks is developed. |
| Approach: | They propose a scalable method for estimating the noisiness of labels produced by crowdsourcing semantic annotation tasks and reducing the resulting error by 20-30%. |
| Outcome: | The proposed method reduces the error of the labeling process by 20-30% compared to other common labeling strategies. |
Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent methods have obtained promising results by extracting relation labels from participants . obtaining linguistic annotations from novice crowdworkers is difficult . crowdsourcing allows for fast and cost-effective collection of labelled data, but because tasks need to be intuitive, crowdworker cannot be asked to perform them. |
| Approach: | They propose to use a selection-only approach to obtain linguistic annotations from novices . current study shows that the method is cost- and time-intensive . |
| Outcome: | The current study shows that selection and training improves the agreement between workers and gold labels, but the method is cost- and time-intensive. |
Crowdsourcing Natural Language Data at Scale: A Hands-On Tutorial (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | a tutorial on crowdsourcing for efficient data annotation will introduce crowdsourcing and provide an overview of the technology. |
| Approach: | This tutorial will introduce users to efficient data annotation via crowdsourcing marketplaces. |
| Outcome: | This tutorial will introduce users to the use of crowdsourcing for data annotation. |