Computer Assisted Annotation of Tension Development in TED Talks through Crowdsourcing (D19-59)
Copied to clipboard
| Challenge: | Using a neural network, we annotate whether tension is increasing, decreasing, or staying unchanged. |
| Approach: | They propose a machine-assisted method for the identification of tension development using a neural network based prediction model. |
| Outcome: | The proposed method is compared with other methods in in-house and crowdsourced environments. |
Similar Papers
A Web-based System for Crowd-in-the-Loop Dependency Treebanking (L18-1)
Copied to clipboard
| Challenge: | Existing treebanks are limited in size, genre, and topic coverage, making manual annotation time-consuming and expensive. |
| Approach: | They propose a web-based interactive tool for editing dependency trees that uses machine learning to accelerate annotation. |
| Outcome: | CROWDTREE is a web-based interactive tool for editing dependency trees . it can train a parsing model during the annotation process and can even be compatible with Mechanical Turk. |
Do you Feel Certain about your Annotation? A Web-based Semantic Frame Annotation Tool Considering Annotators’ Concerns and Behaviors (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing tools for manual annotations are resourceintensive and complex, and experienced annotators and tools specialized for the purpose of the annotation task are required. |
| Approach: | They propose to use a web-based application with a responsive design for modular semantic frame annotation (SFA) the proposed application keeps track of the time and changes during the annotation process and stores the users’ confidence with the current annotation. |
| Outcome: | The proposed system can be used to build a manually annotated corpus and its arguments for task 2 of SemEval 2019 regarding unsupervised lexical frame induction. |
Neural Dependency Parsing of Biomedical Text: TurkuNLP entry in the CRAFT Structural Annotation Task (D19-57)
Copied to clipboard
| Challenge: | Syntactic analysis (parsing) is a fundamental task in natural language processing (NLP). |
| Approach: | They propose to use the Turku neural parser to adapt it to the biomedical domain . they evaluated custom word embeddings, combination with other in-domain resources . |
| Outcome: | The proposed approach achieved a labeled attachment score of 89.7%, the best among task participants. |
Dependency Tree Annotation with Mechanical Turk (D19-59)
Copied to clipboard
| Challenge: | a recent study shows that crowdsourcing is often used to obtain linguistic annotations but is rarely used for parsing. |
| Approach: | They propose to use Mechanical Turk to crowdsource parse trees using an interactive graphical dependency tree editor. |
| Outcome: | The proposed method is the first published use of Mechanical Turk to crowdsource parse trees . the authors find that the workers achieve high levels of accuracy on 72% of the sentences . |
Identifying and Resolving Annotation Changes for Natural Language Understanding (2021.naacl-industry)
Copied to clipboard
| Challenge: | Annotation conflict resolution is crucial for machine learning, says a new study . past work on annotation conflict resolution assumed data is collected at once . a a supervised neural model can resolve conflicts in data annotation but requires access to high-quality data . |
| Approach: | They propose an approach to resolve annotation conflicts in a real-world context using a German dialog system. |
| Outcome: | The proposed approach improves on a real-world dataset with 3.5M utterances in German. |
Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent methods have obtained promising results by extracting relation labels from participants . obtaining linguistic annotations from novice crowdworkers is difficult . crowdsourcing allows for fast and cost-effective collection of labelled data, but because tasks need to be intuitive, crowdworker cannot be asked to perform them. |
| Approach: | They propose to use a selection-only approach to obtain linguistic annotations from novices . current study shows that the method is cost- and time-intensive . |
| Outcome: | The current study shows that selection and training improves the agreement between workers and gold labels, but the method is cost- and time-intensive. |
Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP (D19-59)
Copied to clipboard
| Challenge: | The first workshop on crowdsourcing for NLP is open to all . |
| Approach: | The first workshop on crowdsourcing annotations for NLP is held at the acl.com . the workshop will focus on methods for aggregating and analysing crowdsourced data for Nl-specific tasks. |
| Outcome: | The first workshop on crowdsourcing for NLP received 16 submissions and accepted 7 . the workshop will focus on ambiguous, subjective or ambiguity analysis of crowdsourced data . |
Analysis of Automatic Annotation Suggestions for Hard Discourse-Level Tasks in Expert Domains (P19-1)
Copied to clipboard
Claudia Schulz, Christian M. Meyer, Jan Kiesewetter, Michael Sailer, Elisabeth Bauer, Martin R. Fischer, Frank Fischer, Iryna Gurevych
| Challenge: | Existing deep learning methods require large amounts of training data to achieve reasonable performance. |
| Approach: | They propose to generate automatic annotation suggestions for a discourse-level sequence labelling task that requires extensive domain expertise. |
| Outcome: | The proposed model improves with newly annotated texts while introducing no biases. |
Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types (C18-1)
Copied to clipboard
| Challenge: | Annotated corpora are often assigned to internet workers whose judgments are reconciled by crowdsourcing models. |
| Approach: | They propose a framework for learning from rich prior knowledge to combine annotations with different structures. |
| Outcome: | The proposed model compares favorably with previous work and enables active sample selection to reduce annotation effort. |
Crowd-sourcing annotation of complex NLU tasks: A case study of argumentative content annotation (D19-59)
Copied to clipboard
| Challenge: | Recent advances in machine reading and listening comprehension involve the annotation of long texts. |
| Approach: | They propose a way to perform a sentence-by-sentence annotation task with crowd annotators. |
| Outcome: | The proposed approach can be used to identify claims in a debate speech. |