Why Don’t You Do It Right? Analysing Annotators’ Disagreement in Subjective Tasks (2023.eacl-main)
Copied to clipboard
| Challenge: | Disagreement can reflect different aspects of linguistic annotation, from annotators’ subjectivity to sloppiness or lack of context to interpret a text. |
| Approach: | They propose a taxonomy of possible reasons leading to annotators' disagreement in subjective tasks and manually label part of a Twitter dataset for offensive language detection in english following this taxonomies. |
| Outcome: | The proposed taxonomy of disagreements in linguistic datasets can be used to assess how accurate tweets belonging to different disagreement categories can be classified as offensive or not. |
Similar Papers
Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement (2021.emnlp-main)
Copied to clipboard
| Challenge: | supervised learning is a key component of offensive language detection, but there is little attention given to the quality of annotated data. |
| Approach: | They propose to examine the level of agreement among annotators while selecting data to create offensive language datasets, a task involving a high level of subjectivity. |
| Outcome: | The proposed datasets show that annotators' agreement has a strong effect on classifiers performance and robustness. |
You Are What You Annotate: Towards Better Models through Annotator Representations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Annotator disagreement is ubiquitous in natural language processing tasks. |
| Approach: | They propose to model annotators' idiosyncrasies and account for their idioms by creating representations for each annotator and their annotations. |
| Outcome: | The proposed model improves on an existing dataset with eight annotators with inherent disagreements while increasing model size by 1%. |
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2022.tacl-1)
Copied to clipboard
| Challenge: | Annotators may systematically disagree with one another, reflecting their individual biases and values, especially in the case of subjective tasks such as detecting affect, aggression, and hate speech. |
| Approach: | They propose to combine multi-annotator models with multi-task based approaches to resolve disagreements between annotations and derive single ground truth labels. |
| Outcome: | The proposed model outperforms majority voting and averaging methods and estimates uncertainty in predictions. |
When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | a number of studies have questioned assumptions of majority vote aggregated labels. |
| Approach: | They construct a model that predicts individual annotator ratings on potentially offensive text and combines this information with the predicted target group of the text to predict the ratings of target group members. |
| Outcome: | The proposed model raises performance over baseline by 22% and 33% at predicting variance among annotators. |
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels (2024.naacl-long)
Copied to clipboard
| Challenge: | a line of recent work has illustrated that annotators disagree for many reasons . capturing disagreements can improve model performance and calibration, authors argue . |
| Approach: | They propose a new paradigm shift in data labeling for machine learning that challenges annotator disagreement by treating disagreement as a valuable source of information. |
| Outcome: | The proposed approaches challenge annotator disagreement and provide recommendations for the data labeling pipeline and avenues for future research. |
Investigating Reasons for Disagreement in Natural Language Inference (2022.tacl-1)
Copied to clipboard
| Challenge: | Several disagreements in natural language inference (NLI) annotation are due to uncertainty in the sentence meaning, others to annotator biases and task artifacts. |
| Approach: | They propose a 4-way classification approach and a multilabel classification approach for detecting disagreements in natural language inference annotations. |
| Outcome: | The proposed model is more expressive and gives better recall of possible interpretations in the data. |
Which Examples Should be Multiply Annotated? Active Learning When Annotators May Disagree (2023.findings-acl)
Copied to clipboard
| Challenge: | Disagreement in annotations is natural for humans, depending on background, identity, positionality . many active learning approaches focus on examples where model entropy and annotator entropicy are the most different. |
| Approach: | They propose an active learning approach that focuses annotations on examples where model entropy and annotator entropic are the most different. |
| Outcome: | The proposed approach reduces the number of annotations required by 24% on average across datasets. |
Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on evaluating large language models' ability to handle disagreement cases. |
| Approach: | They evaluate the performance of large language models in detecting offensive language at varying levels of agreement. |
| Outcome: | The proposed model improves detection accuracy and model alignment with human judgment by using disagreement samples in training. |
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets (D19-1)
Copied to clipboard
| Challenge: | Having only a few workers generate the majority of dataset examples raises concerns about data diversity . |
| Approach: | They perform a series of experiments to investigate annotator biases in recent NLU datasets . they find that models are able to recognize the most productive annotators . |
| Outcome: | The results show that models can recognize the most productive annotators and do not generalize well to examples from annotator that did not contribute to the training set. |
Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Annotator disagreements are resolved before learning takes place, but researchers question the performance of a system when annotators disagree. |
| Approach: | They propose a method that uses language features and label distributions to pool similar items into larger labels. |
| Outcome: | The proposed method is based on five publicly available datasets with varying levels of disagreements on social media and in the wild using a dataset from Facebook. |