The Ecological Fallacy in Annotation: Modeling Human Label Variation goes beyond Sociodemographics (2023.acl-short)
Copied to clipboard
| Challenge: | Existing work has attempted to model individual annotation behaviour rather than predicting aggregated labels. |
| Approach: | They propose to model individual annotator behaviour rather than predicting aggregated labels by adding group-specific layers to multi-annotator models to account for sociodemographics. |
| Outcome: | The proposed model does not significantly improve on toxic content detection tasks. |
Similar Papers
Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals’ Subjective Text Perceptions (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that LLMs perform poorly when prompted with sociodemographic attributes, suggesting limited inherent sociodemography knowledge. |
| Approach: | They propose to train large language models to be accurate sociodemographic models of annotator variation by using a curated dataset of five tasks with standardized sociodemography. |
| Outcome: | The proposed models improve in sociodemographic prompting when trained but this performance gain is largely due to models learning annotator-specific behaviour rather than sociodemography. |
The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | a paper argues that human label variation impacts all stages of the ML pipeline . human label variations are often considered noise due to disagreement, subjectivity in annotation or multiple plausible answers. |
| Approach: | They propose to reconcile different notions of human label variation and propose a repository of publicly-available datasets with un-aggregated labels. |
| Outcome: | The proposed approaches are compared with publicly available datasets with un-aggregated labels and identify gaps. |
Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown considerable promise for annotation purposes, but questions remain about their ability to capture human label variation (HLV) label variation is genuine disagreement between annotators observed across NLP tasks. |
| Approach: | They investigate how label and explanation variation manifests within and across LLMs with respect to the Natural Language Inference task. |
| Outcome: | The proposed models generate label distributions similar to humans but exhibit distinct, idiosyncratic judgments and disagreement patterns. |
Learning Subjective Label Distributions via Sociocultural Descriptors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Conventional approaches aggregate annotator judgements (labels), ignoring minority perspectives, and overlooking the influence of sociocultural context behind such annotations. |
| Approach: | They propose a framework where subjectivity in binary labels is modeled as an empirical distribution accounting for the variation in annotators through human values extracted from sociocultural descriptors using a language model. |
| Outcome: | The proposed model yields well-calibrated toxicity distribution predictions across binary toxicity labels, which are further used for majority label prediction across cultural subgroups. |
Ecologically Valid Explanations for Label Variation in NLI (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Human label variation exists in many natural language processing tasks, including NLI . |
| Approach: | They build an English dataset of 1,415 ecologically valid explanations for 122 MNLI items . they find that people can systematically vary on their interpretation . |
| Outcome: | The proposed dataset contains 1,415 ecologically valid explanations for 122 items . the results show that people can vary on interpretation and highlight differences . |
VariErr NLI: Separating Annotation Error from Human Label Variation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing work on label variation and annotation errors has focused on them in isolation. |
| Approach: | They propose a 2-round annotation procedure to separate human label variation from annotation errors by pairing valid explanations with annotators' validations. |
| Outcome: | The proposed procedure is based on the NLI task in English and contains 7,732 valid judgements on 1,933 explanations for 500 re-annotated items. |
Which Demographics do LLMs Default to During Annotation? (2025.acl-long)
Copied to clipboard
Johannes Schäfer, Aidan Combs, Christopher Bagdon, Jiahui Li, Nadine Probol, Lynn Greschner, Sean Papay, Yarik Menchaca Resendiz, Aswathy Velutharambath, Amelie Wuehrl, Sabine Weber, Roman Klinger
| Challenge: | Demographics and cultural background of annotators influence the labels they assign in text annotation. |
| Approach: | They examine the attributes of human annotators LLMs inherently mimic and compare them to demographic-conditioned prompts and placebo-conditioned ones. |
| Outcome: | The proposed model incorporates demographics and cultural background into the output of the large language models (LLMs) to evaluate which attributes of human annotators LLMs inherently mimic. |
On the Interplay between Human Label Variation and Model Fairness (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on the impact of human label variation on model fairness have not explored the interaction between HLV and performance. |
| Approach: | They compare human label variation (HLV) training methods with four other methods . they find that HLV methods improve performance without harming fairness . |
| Outcome: | The proposed methods improve fairness without explicit debiasing under certain configurations. |
Fine-grained Fallacy Detection with Human Label Variation (2025.naacl-long)
Copied to clipboard
| Challenge: | Fallacy detection is an open challenge in NLP and has shown to be intrinsically difficult for both humans and machines. |
| Approach: | They propose a framework that minimizes annotation errors whilst keeping signals of human label variation. |
| Outcome: | The proposed framework minimizes annotation errors while keeping signals of human label variation. |
Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing studies on sociodemographic prompting have not explored the effectiveness of this technique. |
| Approach: | They propose to use sociodemographic prompting to steer models towards answers that humans with specific sociodemography would give. |
| Outcome: | The proposed technique can improve zero-shot learning by focusing on human sociodemographic profiles. |