On the Interplay between Human Label Variation and Model Fairness (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on the impact of human label variation on model fairness have not explored the interaction between HLV and performance. |
| Approach: | They compare human label variation (HLV) training methods with four other methods . they find that HLV methods improve performance without harming fairness . |
| Outcome: | The proposed methods improve fairness without explicit debiasing under certain configurations. |
Similar Papers
Position: From Noise to Signal to Selbstzweck - Reframing Human Label Variation in the Era of Post-training in NLP (2026.findings-acl)
Copied to clipboard
| Challenge: | Human Label Variation (HLV) refers to legitimate disagreement in annotation . current preference-learning datasets routinely collapse multiple annotations into a single label . |
| Approach: | They propose to preserve human label variation as an embodiment of pluralism . they argue that disagreement in annotations should be treated as a selfzweck . |
| Outcome: | The proposed approach preserves pluralism and human pluralismos, the authors argue . they argue that disagreements in annotations should be treated as a selfzweck . |
Mining the uncertainty patterns of humans and models in the annotation of moral foundations and human values (2025.acl-long)
Copied to clipboard
| Challenge: | disagreement in annotation (HLV) is considered a constitutive feature of subjective tasks. |
| Approach: | They investigate the relationship between disagreement in annotation and model uncertainty . they use linguistic features to calibrate models to HLV and uncertainty to analyze their impact on uncertainty. |
| Outcome: | The proposed model uncertainty is calibrated to human label variation (HLV) the proposed model is calibrate to human labels, the authors show . |
The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | a paper argues that human label variation impacts all stages of the ML pipeline . human label variations are often considered noise due to disagreement, subjectivity in annotation or multiple plausible answers. |
| Approach: | They propose to reconcile different notions of human label variation and propose a repository of publicly-available datasets with un-aggregated labels. |
| Outcome: | The proposed approaches are compared with publicly available datasets with un-aggregated labels and identify gaps. |
Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown considerable promise for annotation purposes, but questions remain about their ability to capture human label variation (HLV) label variation is genuine disagreement between annotators observed across NLP tasks. |
| Approach: | They investigate how label and explanation variation manifests within and across LLMs with respect to the Natural Language Inference task. |
| Outcome: | The proposed models generate label distributions similar to humans but exhibit distinct, idiosyncratic judgments and disagreement patterns. |
“Seeing the Big through the Small”: Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Human label variation arises when multiple human annotators provide different labels for valid reasons. |
| Approach: | They propose to use crowd workers to represent human judgment distributions or expert linguists to provide detailed explanations for their chosen labels. |
| Outcome: | The proposed model can approximate human judgment distributions using a small number of expert labels and explanations. |
Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback (2024.acl-long)
Copied to clipboard
| Challenge: | a growing body of work on learning from human feedback to align various aspects of machine learning systems with human values and preferences is focusing on the setting of fairness in content moderation. |
| Approach: | They propose to use human feedback to determine how two comments should be treated in content moderation to learn about human values and preferences. |
| Outcome: | The proposed approach is promising, as human preferences can often not be A: Some ladies like smaller men. B: Some men like smaller guys. Figure 1 shows that the proposed approach performs better for demographic intersections than a single classifier that gives equal weight to each annotation. |
The Ecological Fallacy in Annotation: Modeling Human Label Variation goes beyond Sociodemographics (2023.acl-short)
Copied to clipboard
| Challenge: | Existing work has attempted to model individual annotation behaviour rather than predicting aggregated labels. |
| Approach: | They propose to model individual annotator behaviour rather than predicting aggregated labels by adding group-specific layers to multi-annotator models to account for sociodemographics. |
| Outcome: | The proposed model does not significantly improve on toxic content detection tasks. |
Interpreting Predictive Probabilities: Model Confidence or Human Label Variation? (2024.eacl-short)
Copied to clipboard
| Challenge: | In modern NLP, neural networks are the de-facto standard to predict complex probability measures from available context. |
| Approach: | They propose to use a single predictive distribution to evaluate models with disentangled representations of uncertainty about predictions and uncertainty about human labels. |
| Outcome: | The proposed models are crucial for trustworthy and fair NLP systems, but exploiting a single distribution is limiting. |
Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models can lead to undesired consequences when misaligned with human values . previous studies have shown misalignment of LLMs with human value using expert-designed or agent-based emulated bias scenarios . |
| Approach: | They investigate whether large language models (LLMs) are misaligned with human values . they find no significant differences in understanding of HVSB between LLMs . |
| Outcome: | The results show that large language models do not have lower misalignment rates and attack success rates . the study also shows that smaller language models have the ability to explain HVSB . |
Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals’ Subjective Text Perceptions (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that LLMs perform poorly when prompted with sociodemographic attributes, suggesting limited inherent sociodemography knowledge. |
| Approach: | They propose to train large language models to be accurate sociodemographic models of annotator variation by using a curated dataset of five tasks with standardized sociodemography. |
| Outcome: | The proposed models improve in sociodemographic prompting when trained but this performance gain is largely due to models learning annotator-specific behaviour rather than sociodemography. |