| Challenge: | Obtaining annotations from experts is ideal, but this expertise is logistically and financially costly. |
| Approach: | They propose an annotation framework that supports the whole annotation pipeline from understanding the resources required for an annotation task to compiling the annotated dataset. |
| Outcome: | The proposed framework improves classification performance through annotator-reliability-based soft-label aggregation and sample weighting, and increases agreement among annotators through removal of identifying and replacing an unreliable annotation. |
Similar Papers
Efficient Annotator Reliability Assessment and Sample Weighting for Knowledge-Based Misinformation Detection on Social Media (2025.findings-naacl)
Copied to clipboard
Owen Cook, Charlie Grimshaw, Ben Peng Wu, Sophie Dillon, Jack Hicks, Luke Jones, Thomas Smith, Matyas Szert, Xingyi Song
| Challenge: | Misinformation spreads rapidly on social media, confusing the truth and targeting potentially vulnerable people. |
| Approach: | They propose to use inter- and intra-annotator agreement to understand the reliability of each annotator and influence the training of large language models based on annotators reliability. |
| Outcome: | The proposed framework utilises inter- and intra-annotator agreement to understand the reliability of each annotator and influence the training of large language models based on annotators reliability. |
FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Existing tools to evaluate long text outputs are lacking in the field of NLP . human rating and error analysis remains a crucial component for any evaluation of long text generation. |
| Approach: | They propose a web-based toolkit to collect fine-grained error annotations for long texts . they use a taxonomy to identify errors and assign them to text spans . |
| Outcome: | The proposed tool can be used to evaluate the coherence of long generated summaries. |
Modelling Instance-Level Annotator Reliability for Natural Language Labelling Tasks (N19-1)
Copied to clipboard
| Challenge: | Existing models that estimate annotators' reliability only consider binary labels and multi-class labels. |
| Approach: | They propose an unsupervised model which can handle binary and multi-class labels and integrate neural networks to model the dependency between latent variables and instances. |
| Outcome: | The proposed model can handle binary and multi-class labels and can estimate reliability of annotators across instances. |
APLenty: annotation tool for creating high-quality datasets using active and proactive learning (D18-2)
Copied to clipboard
| Challenge: | APLenty is an annotation tool for creating high-quality sequence labeling datasets using active and proactive learning. |
| Approach: | They present APLenty, an annotation tool for creating high-quality sequence labeling datasets using active and proactive learning. |
| Outcome: | The proposed tool is highly flexible and can be adapted to various other tasks. |
A Web-based Collaborative Annotation and Consolidation Tool (2020.lrec-1)
Copied to clipboard
| Challenge: | Annotation tools have a rigid structure, closed back-end and front-end, and are built in a non-user-friendly way rendering them unusable for a large cohort. |
| Approach: | They propose a web-based collaborative annotation and consolidation tool (AWOCATo) that supports varied textual formats and allows users to easily adapt to the annotation task. |
| Outcome: | AWOCATo supports a range of tasks and domains, filling the gap left by the lack of tools that can be used by people with and without programming knowledge. |
Easy, Reproducible and Quality-Controlled Data Collection with CROWDAQ (2020.emnlp-demos)
Copied to clipboard
Qiang Ning, Hao Wu, Pradeep Dasigi, Dheeru Dua, Matt Gardner, Robert L. Logan IV, Ana Marasović, Zhen Nie
| Challenge: | Efficient data collection is important for advancing research and building time-sensitive applications. |
| Approach: | They propose an open-source platform that standardizes the data collection pipeline . it includes customizable user interface components, automated annotator qualification, and saved pipelines . |
| Outcome: | The proposed platform simplifies data annotation significantly on diverse datasets . it can be used by researchers and engineers to improve reproducibility and minimize overhead . |
KCAT: A Knowledge-Constraint Typing Annotation Tool (P19-3)
Copied to clipboard
Sheng Lin, Luye Zheng, Bo Chen, Siliang Tang, Zhigang Chen, Guoping Hu, Yueting Zhuang, Fei Wu, Xiang Ren
| Challenge: | Recent years Natural Language Processing community has seen a surge of interest in fine-grained entity typing (FET) given an entity mention (i.e. a sequence of token spans representing an entity), FET aims at uncovering its contextdependent type. |
| Approach: | They propose an efficient Knowledge Constraint Fine-grained Entity Typing Annotation Tool which further improves the entity typing process through entity linking together with some practical functions. |
| Outcome: | The proposed tool improves the entity typing process by linking the candidate types with some practical functions. |
Corpus Considerations for Annotator Modeling and Scaling (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent trends in natural language processing and annotation tasks emphasize individual perspectives . annotator models that rely on a single ground truth may disregard valuable minority perspectives omissions . |
| Approach: | They propose a composite embedding approach to investigate annotator modeling techniques . they show that the commonly used user token model consistently outperforms more complex models . |
| Outcome: | The proposed model outperforms more complex models on a given dataset. |
Errator: a Tool to Help Detect Annotation Errors in the Universal Dependencies Project (L18-1)
Copied to clipboard
| Challenge: | UD project aims to develop cross-linguistically consistent treebank annotations for a wide array of languages. |
| Approach: | They introduce tools that implement the annotation variation principle to help annotators find and correct errors in UD treebanks. |
| Outcome: | The proposed tools can be used to correct errors in UD treebank annotations. |
Don’t waste a single annotation: improving single-label classifiers through soft labels (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for annotating data are limited by ambiguity and lack of context in data samples. |
| Approach: | They challenge the traditional approach of annotating data by only providing a single label for each sample and annotator disagreement is discarded . instead, they use additional annotation information such as confidence, secondary label and disagreement to generate soft labels. |
| Outcome: | The proposed method improves model performance and calibration on the hard label test set. |