| Challenge: | Existing methods to generate annotated corpora for coreference are expensive and limited. |
| Approach: | They propose a model of annotation for aggregating crowdsourced anaphoric annotations. |
| Outcome: | The proposed model can extract from crowdsourced annotations coreference chains comparable to those obtained with expert annotation. |
Similar Papers
A Neural Model for Aggregating Coreference Annotation in Crowdsourcing (2020.coling-main)
Copied to clipboard
| Challenge: | Existing studies of natural language labelling tasks have shown that crowd-sourced labels can be noisy. |
| Approach: | They split the aggregation into mention classification and coreference chain inference tasks to predict the correct labels. |
| Outcome: | The proposed model predicts the class of each mention using an autoencoder while taking into account the mention’s annotation complexity and annotators’ reliability at different levels. |
A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation (N19-1)
Copied to clipboard
| Challenge: | a corpus of anaphoric information (coreference) is crowdsourced through a game-with-a-purpose . its main feature is the large number of judgments per markable: 20 on average, and over 2.2M in total. |
| Approach: | They propose to crowdsource anaphoric information corpus by a game-with-a-purpose and to use it to train a coreference resolver. |
| Outcome: | The proposed corpus contains annotations for 108,000 markables and 20 judgments per markable, and 2.2M in total. |
ezCoref: Towards Unifying Annotation Guidelines for Coreference Resolution (2023.findings-eacl)
Copied to clipboard
Ankita Gupta, Marzena Karpinska, Wenlong Zhao, Kalpesh Krishna, Jack Merullo, Luke Yeh, Mohit Iyyer, Brendan O’Connor
| Challenge: | Existing datasets vary in definition of coreferences and are curated for linguistic experts. |
| Approach: | They propose to use ezCoref to create a crowdsourcing-friendly coreference annotation methodology that teaches annotators only cases that are treated similarly across existing datasets. |
| Outcome: | The proposed method reannotates 240 passages from seven existing english coreference datasets while teaching annotators only cases that are treated similarly across them. |
Model-based Annotation of Coreference (2020.lrec-1)
Copied to clipboard
| Challenge: | Annotators are asked to annotate coreferent spans of text, which is unnatural . we present an alternative in which annotators can preprocess documents and assign pronouns to entities. |
| Approach: | They propose an alternative in which annotators are asked to assign pronouns to entities and preprocess documents to create a knowledge base. |
| Outcome: | The proposed model-based approach leads to faster annotation and higher inter-annotator agreement and opens up an alternative approach to coreference resolution. |
Active Learning for Coreference Resolution using Discrete Annotation (2020.acl-main)
Copied to clipboard
| Challenge: | Exhaustively annotating coreference is expensive as it requires tracking coreference chains across long passages of text. |
| Approach: | They propose a pairwise annotation technique which asks annotators to identify mention antecedents if a presented mention pair is not coreferent. |
| Outcome: | The proposed method is much more efficient when combined with a mention clustering algorithm for selecting which examples to label . future work can use the proposed protocol to develop coreference models for new domains. |
Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types (C18-1)
Copied to clipboard
| Challenge: | Annotated corpora are often assigned to internet workers whose judgments are reconciled by crowdsourcing models. |
| Approach: | They propose a framework for learning from rich prior knowledge to combine annotations with different structures. |
| Outcome: | The proposed model compares favorably with previous work and enables active sample selection to reduce annotation effort. |
Adapting Coreference Resolution Models through Active Learning (2022.acl-long)
Copied to clipboard
| Challenge: | Neural coreference resolution models trained on one dataset may not transfer to new, low-resource domains. |
| Approach: | They investigate how to actively label coreference by sampling a small subset of data for annotators to label. |
| Outcome: | The proposed model can be more realistic when labeling spans within the same document than when annotating spans across documents. |
Corpus Considerations for Annotator Modeling and Scaling (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent trends in natural language processing and annotation tasks emphasize individual perspectives . annotator models that rely on a single ground truth may disregard valuable minority perspectives omissions . |
| Approach: | They propose a composite embedding approach to investigate annotator modeling techniques . they show that the commonly used user token model consistently outperforms more complex models . |
| Outcome: | The proposed model outperforms more complex models on a given dataset. |
Aggregating Crowdsourced and Automatic Judgments to Scale Up a Corpus of Anaphoric Reference for Fiction and Wikipedia Texts (2023.eacl-main)
Copied to clipboard
Juntao Yu, Silviu Paun, Maris Camilleri, Paloma Garcia, Jon Chamberlain, Udo Kruschwitz, Massimo Poesio
| Challenge: | Existing approaches to scale up anaphoric annotation have not overcome these limitations. |
| Approach: | They propose to use a game-with-a-purpose to ‘complete’ markable annotations by using an anaphoric resolver and an aggregation method for anaphorism. |
| Outcome: | The proposed method could be adopted to greatly speed up annotation time in other projects involving games-with-a-purpose. |
ParCorFull: a Parallel Corpus Annotated with Full Coreference (L18-1)
Copied to clipboard
| Challenge: | Recent research in multilingual coreference and automatic pronoun translation has led to important insights into the problem and some promising results. |
| Approach: | They propose a corpus annotated with full coreference chains that addresses a problem that machine translation and other multilingual natural language processing (NLP) technologies face: translation of coreference across languages. |
| Outcome: | The proposed corpus contains parallel texts for the language pair English-German, two major European languages. |