The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels (2024.naacl-long)
Copied to clipboard
| Challenge: | a line of recent work has illustrated that annotators disagree for many reasons . capturing disagreements can improve model performance and calibration, authors argue . |
| Approach: | They propose a new paradigm shift in data labeling for machine learning that challenges annotator disagreement by treating disagreement as a valuable source of information. |
| Outcome: | The proposed approaches challenge annotator disagreement and provide recommendations for the data labeling pipeline and avenues for future research. |
Similar Papers
Why Don’t You Do It Right? Analysing Annotators’ Disagreement in Subjective Tasks (2023.eacl-main)
Copied to clipboard
| Challenge: | Disagreement can reflect different aspects of linguistic annotation, from annotators’ subjectivity to sloppiness or lack of context to interpret a text. |
| Approach: | They propose a taxonomy of possible reasons leading to annotators' disagreement in subjective tasks and manually label part of a Twitter dataset for offensive language detection in english following this taxonomies. |
| Outcome: | The proposed taxonomy of disagreements in linguistic datasets can be used to assess how accurate tweets belonging to different disagreement categories can be classified as offensive or not. |
Unifying Data Perspectivism and Personalization: An Application to Social Norms (2022.emnlp-main)
Copied to clipboard
| Challenge: | Obtaining a single ground truth is not possible or necessary for subjective tasks. |
| Approach: | They propose a set of personalization methods to model annotators and compare their effectiveness for predicting social norms. |
| Outcome: | The proposed model outperforms existing models and compares performance across subsets of social situations that vary by the closeness of the relationship between parties in conflict. |
Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning (2021.naacl-main)
Copied to clipboard
| Challenge: | Prior work shows that disagreement between annotators can be useful in training models. |
| Approach: | They propose to use disagreements as an auxiliary task in a multi-task neural network to incorporate disagreements into models. |
| Outcome: | The proposed method significantly improves performance on NLP tasks beyond the standard approach and prior work. |
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2022.tacl-1)
Copied to clipboard
| Challenge: | Annotators may systematically disagree with one another, reflecting their individual biases and values, especially in the case of subjective tasks such as detecting affect, aggression, and hate speech. |
| Approach: | They propose to combine multi-annotator models with multi-task based approaches to resolve disagreements between annotations and derive single ground truth labels. |
| Outcome: | The proposed model outperforms majority voting and averaging methods and estimates uncertainty in predictions. |
You Are What You Annotate: Towards Better Models through Annotator Representations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Annotator disagreement is ubiquitous in natural language processing tasks. |
| Approach: | They propose to model annotators' idiosyncrasies and account for their idioms by creating representations for each annotator and their annotations. |
| Outcome: | The proposed model improves on an existing dataset with eight annotators with inherent disagreements while increasing model size by 1%. |
Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks (2022.naacl-main)
Copied to clipboard
| Challenge: | Labelled data is the foundation of most natural language processing tasks, but there are valid beliefs about what the correct data labels should be. |
| Approach: | They propose two contrasting paradigms for data annotation that encourage annotator subjectivity . they propose a descriptive paradigm that allows for the surveying and modelling of different beliefs . |
| Outcome: | The proposed paradigms encourage annotator subjectivity, while the prescriptive paradigm discourages it. |
Disagreement Matters: Preserving Label Diversity by Jointly Modeling Item and Annotator Label Distributions with DisCo (2023.findings-acl)
Copied to clipboard
Tharindu Cyril Weerasooriya, Alexander Ororbia, Raj Bhensadadia, Ashiqur KhudaBukhsh, Christopher Homan
| Challenge: | a recent study shows that annotator disagreement is common in supervised learning . a simple neural model that learns to predict annotators' labels is competitive with other models that do not model specific annotations. |
| Approach: | They propose a neural model that learns to predict annotator distributions by aggregating over all annotators. |
| Outcome: | The proposed model outperforms models that do not model specific annotators or do not learn label distribution learning. |
PERSEVAL: A Framework for Perspectivist Classification Evaluation (2025.emnlp-main)
Copied to clipboard
Soda Marem Lo, Silvia Casola, Erhan Sezerer, Valerio Basile, Franco Sansonetti, Antonio Uva, Davide Bernardi
| Challenge: | Perspectivist evaluation practices in NLP remain fragmented and inconsistent . |
| Approach: | They propose a framework that evaluates perspectivist models at the individual annotator level and treats annotators and users as distinct entities, consistent with real-world scenarios. |
| Outcome: | The proposed framework evaluates annotators and users as distinct entities consistent with real-world scenarios. |
Investigating Reasons for Disagreement in Natural Language Inference (2022.tacl-1)
Copied to clipboard
| Challenge: | Several disagreements in natural language inference (NLI) annotation are due to uncertainty in the sentence meaning, others to annotator biases and task artifacts. |
| Approach: | They propose a 4-way classification approach and a multilabel classification approach for detecting disagreements in natural language inference annotations. |
| Outcome: | The proposed model is more expressive and gives better recall of possible interpretations in the data. |
The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | a paper argues that human label variation impacts all stages of the ML pipeline . human label variations are often considered noise due to disagreement, subjectivity in annotation or multiple plausible answers. |
| Approach: | They propose to reconcile different notions of human label variation and propose a repository of publicly-available datasets with un-aggregated labels. |
| Outcome: | The proposed approaches are compared with publicly available datasets with un-aggregated labels and identify gaps. |