Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Annotator disagreements are resolved before learning takes place, but researchers question the performance of a system when annotators disagree. |
| Approach: | They propose a method that uses language features and label distributions to pool similar items into larger labels. |
| Outcome: | The proposed method is based on five publicly available datasets with varying levels of disagreements on social media and in the wild using a dataset from Facebook. |
Similar Papers
Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to label aggregation fail to capture subjective annotations and can lead to biases. |
| Approach: | They propose annotator-aware representations for text for subjective classification tasks that involve learning representations of annotators. |
| Outcome: | The proposed model improves on metrics that assess the performance on capturing individual annotators’ perspectives. |
Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement (2021.emnlp-main)
Copied to clipboard
| Challenge: | supervised learning is a key component of offensive language detection, but there is little attention given to the quality of annotated data. |
| Approach: | They propose to examine the level of agreement among annotators while selecting data to create offensive language datasets, a task involving a high level of subjectivity. |
| Outcome: | The proposed datasets show that annotators' agreement has a strong effect on classifiers performance and robustness. |
When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | a number of studies have questioned assumptions of majority vote aggregated labels. |
| Approach: | They construct a model that predicts individual annotator ratings on potentially offensive text and combines this information with the predicted target group of the text to predict the ratings of target group members. |
| Outcome: | The proposed model raises performance over baseline by 22% and 33% at predicting variance among annotators. |
Voices in a Crowd: Searching for clusters of unique perspectives (2024.emnlp-main)
Copied to clipboard
| Challenge: | Proposed solutions aim to capture minority perspectives by either modelling annotator disagreements or grouping annotators based on shared metadata. |
| Approach: | They propose a framework that trains models without encoding annotator metadata and creates clusters of similar opinions, that are called voices. |
| Outcome: | The proposed framework captures minority perspectives based on demographic factors in two distinct datasets while also capturing majority perspectives. |
Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types (C18-1)
Copied to clipboard
| Challenge: | Annotated corpora are often assigned to internet workers whose judgments are reconciled by crowdsourcing models. |
| Approach: | They propose a framework for learning from rich prior knowledge to combine annotations with different structures. |
| Outcome: | The proposed model compares favorably with previous work and enables active sample selection to reduce annotation effort. |
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2022.tacl-1)
Copied to clipboard
| Challenge: | Annotators may systematically disagree with one another, reflecting their individual biases and values, especially in the case of subjective tasks such as detecting affect, aggression, and hate speech. |
| Approach: | They propose to combine multi-annotator models with multi-task based approaches to resolve disagreements between annotations and derive single ground truth labels. |
| Outcome: | The proposed model outperforms majority voting and averaging methods and estimates uncertainty in predictions. |
Disagreement Matters: Preserving Label Diversity by Jointly Modeling Item and Annotator Label Distributions with DisCo (2023.findings-acl)
Copied to clipboard
Tharindu Cyril Weerasooriya, Alexander Ororbia, Raj Bhensadadia, Ashiqur KhudaBukhsh, Christopher Homan
| Challenge: | a recent study shows that annotator disagreement is common in supervised learning . a simple neural model that learns to predict annotators' labels is competitive with other models that do not model specific annotations. |
| Approach: | They propose a neural model that learns to predict annotator distributions by aggregating over all annotators. |
| Outcome: | The proposed model outperforms models that do not model specific annotators or do not learn label distribution learning. |
You Are What You Annotate: Towards Better Models through Annotator Representations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Annotator disagreement is ubiquitous in natural language processing tasks. |
| Approach: | They propose to model annotators' idiosyncrasies and account for their idioms by creating representations for each annotator and their annotations. |
| Outcome: | The proposed model improves on an existing dataset with eight annotators with inherent disagreements while increasing model size by 1%. |
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets (D19-1)
Copied to clipboard
| Challenge: | Having only a few workers generate the majority of dataset examples raises concerns about data diversity . |
| Approach: | They perform a series of experiments to investigate annotator biases in recent NLU datasets . they find that models are able to recognize the most productive annotators . |
| Outcome: | The results show that models can recognize the most productive annotators and do not generalize well to examples from annotator that did not contribute to the training set. |
NUTMEG: Separating Signal From Noise in Annotator Disagreement (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work suggests that annotators may have genuine disagreements, but few models separate signal from noise in annotator disagreement. |
| Approach: | They propose a Bayesian model that removes noisy annotations from training data while preserving systematic disagreements. |
| Outcome: | The proposed model outperforms models trained on NUTMEG-aggregated data. |