A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation (N19-1)
Copied to clipboard
| Challenge: | a corpus of anaphoric information (coreference) is crowdsourced through a game-with-a-purpose . its main feature is the large number of judgments per markable: 20 on average, and over 2.2M in total. |
| Approach: | They propose to crowdsource anaphoric information corpus by a game-with-a-purpose and to use it to train a coreference resolver. |
| Outcome: | The proposed corpus contains annotations for 108,000 markables and 20 judgments per markable, and 2.2M in total. |
Similar Papers
Aggregating Crowdsourced and Automatic Judgments to Scale Up a Corpus of Anaphoric Reference for Fiction and Wikipedia Texts (2023.eacl-main)
Copied to clipboard
Juntao Yu, Silviu Paun, Maris Camilleri, Paloma Garcia, Jon Chamberlain, Udo Kruschwitz, Massimo Poesio
| Challenge: | Existing approaches to scale up anaphoric annotation have not overcome these limitations. |
| Approach: | They propose to use a game-with-a-purpose to ‘complete’ markable annotations by using an anaphoric resolver and an aggregation method for anaphorism. |
| Outcome: | The proposed method could be adopted to greatly speed up annotation time in other projects involving games-with-a-purpose. |
A Probabilistic Annotation Model for Crowdsourcing Coreference (D18-1)
Copied to clipboard
| Challenge: | Existing methods to generate annotated corpora for coreference are expensive and limited. |
| Approach: | They propose a model of annotation for aggregating crowdsourced anaphoric annotations. |
| Outcome: | The proposed model can extract from crowdsourced annotations coreference chains comparable to those obtained with expert annotation. |
ezCoref: Towards Unifying Annotation Guidelines for Coreference Resolution (2023.findings-eacl)
Copied to clipboard
Ankita Gupta, Marzena Karpinska, Wenlong Zhao, Kalpesh Krishna, Jack Merullo, Luke Yeh, Mohit Iyyer, Brendan O’Connor
| Challenge: | Existing datasets vary in definition of coreferences and are curated for linguistic experts. |
| Approach: | They propose to use ezCoref to create a crowdsourcing-friendly coreference annotation methodology that teaches annotators only cases that are treated similarly across existing datasets. |
| Outcome: | The proposed method reannotates 240 passages from seven existing english coreference datasets while teaching annotators only cases that are treated similarly across them. |
A Neural Model for Aggregating Coreference Annotation in Crowdsourcing (2020.coling-main)
Copied to clipboard
| Challenge: | Existing studies of natural language labelling tasks have shown that crowd-sourced labels can be noisy. |
| Approach: | They split the aggregation into mention classification and coreference chain inference tasks to predict the correct labels. |
| Outcome: | The proposed model predicts the class of each mention using an autoencoder while taking into account the mention’s annotation complexity and annotators’ reliability at different levels. |
Crowdsourcing and Aggregating Nested Markable Annotations (P19-1)
Copied to clipboard
| Challenge: | Existing methods for identifying markables for coreference annotation are task and language-independent and can be used for a variety of other annotation tasks. |
| Approach: | They propose a method for identifying markables for coreference annotation that combines automatic markable detectors with checking with a Game-With-A-Purpose (GWAP) and aggregation using a Bayesian annotation model. |
| Outcome: | The proposed method improves mention boundaries on news and other genres by over seven percentage points compared with state-of-the-art, domain-independent automatic mention detectors and almost three points over an in-domain mention detector. |
ParCorFull: a Parallel Corpus Annotated with Full Coreference (L18-1)
Copied to clipboard
| Challenge: | Recent research in multilingual coreference and automatic pronoun translation has led to important insights into the problem and some promising results. |
| Approach: | They propose a corpus annotated with full coreference chains that addresses a problem that machine translation and other multilingual natural language processing (NLP) technologies face: translation of coreference across languages. |
| Outcome: | The proposed corpus contains parallel texts for the language pair English-German, two major European languages. |
A Crowdsourced Frame Disambiguation Corpus with Ambiguity (N19-1)
Copied to clipboard
| Challenge: | Using crowdsourcing, we have found that inter-annotator disagreement is at least partly caused by ambiguity inherent to the text and frames. |
| Approach: | They propose a crowdsourcing approach to capture inter-annotator disagreement by a list of frames with disagreement-based scores that express the confidence with which each frame applies to the word. |
| Outcome: | The proposed approach captures disagreement between the annotations of 1,000 word-sentence pairs and scores on the likelihood that each frame applies to the word. |
Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types (C18-1)
Copied to clipboard
| Challenge: | Annotated corpora are often assigned to internet workers whose judgments are reconciled by crowdsourcing models. |
| Approach: | They propose a framework for learning from rich prior knowledge to combine annotations with different structures. |
| Outcome: | The proposed model compares favorably with previous work and enables active sample selection to reduce annotation effort. |
Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP (D19-59)
Copied to clipboard
| Challenge: | The first workshop on crowdsourcing for NLP is open to all . |
| Approach: | The first workshop on crowdsourcing annotations for NLP is held at the acl.com . the workshop will focus on methods for aggregating and analysing crowdsourced data for Nl-specific tasks. |
| Outcome: | The first workshop on crowdsourcing for NLP received 16 submissions and accepted 7 . the workshop will focus on ambiguous, subjective or ambiguity analysis of crowdsourced data . |
SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization (D19-54)
Copied to clipboard
| Challenge: | Existing work on abstractive dialogue summarizations has focused on news summarizing but there is no such comprehensive dataset. |
| Approach: | They propose to use a chat-dialogues corpus with abstractive dialogue summaries to generate a short version of text that covers the main points succinctly. |
| Outcome: | The proposed dataset achieves higher ROUGE scores than the model-generated summaries of news, compared with human evaluators' judgement. |