Papers with HLV
Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classification (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for NLP calibration ignore inherent human label variation (HLV) split votes are a problem in high-stakes domains such as legal and medical decisions . |
| Approach: | They present a case outcome classification dataset with judges' vote distributions and build a taxonomy of disagreement with SV-specific subcategories. |
| Outcome: | The proposed model is compared against a judge vote distribution and assesses the alignment of perceived difficulty between models and humans. |
On the Interplay between Human Label Variation and Model Fairness (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on the impact of human label variation on model fairness have not explored the interaction between HLV and performance. |
| Approach: | They compare human label variation (HLV) training methods with four other methods . they find that HLV methods improve performance without harming fairness . |
| Outcome: | The proposed methods improve fairness without explicit debiasing under certain configurations. |
EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI (2026.findings-acl)
Copied to clipboard
| Challenge: | Human label variation (HLV) arises when multiple labels are valid for the same instance. |
| Approach: | They propose a framework for generating and validating explanations to detect errors using large language models (LLMs) EVADE framework provides broader explanation coverage and requires less human intervention . |
| Outcome: | The proposed framework provides broader explanation coverage, requires less human intervention, and delivers better downstream performance in predicting label distributions. |
A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent research has shown that explanations provide valuable information for understanding human label variation (HLV) Large language models (LLMs) can approximate HJD from a few human-provided label-explanation pairs, but collecting explanations for every label is still time-consuming. |
| Approach: | They propose to use Large Language Models (LLMs) as annotators to generate model explanations for a few given human labels. |
| Outcome: | The proposed models can generate human-provided explanations from human labels, but they are still time-consuming. |
Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown considerable promise for annotation purposes, but questions remain about their ability to capture human label variation (HLV) label variation is genuine disagreement between annotators observed across NLP tasks. |
| Approach: | They investigate how label and explanation variation manifests within and across LLMs with respect to the Natural Language Inference task. |
| Outcome: | The proposed models generate label distributions similar to humans but exhibit distinct, idiosyncratic judgments and disagreement patterns. |
“Seeing the Big through the Small”: Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Human label variation arises when multiple human annotators provide different labels for valid reasons. |
| Approach: | They propose to use crowd workers to represent human judgment distributions or expert linguists to provide detailed explanations for their chosen labels. |
| Outcome: | The proposed model can approximate human judgment distributions using a small number of expert labels and explanations. |
Mining the uncertainty patterns of humans and models in the annotation of moral foundations and human values (2025.acl-long)
Copied to clipboard
| Challenge: | disagreement in annotation (HLV) is considered a constitutive feature of subjective tasks. |
| Approach: | They investigate the relationship between disagreement in annotation and model uncertainty . they use linguistic features to calibrate models to HLV and uncertainty to analyze their impact on uncertainty. |
| Outcome: | The proposed model uncertainty is calibrated to human label variation (HLV) the proposed model is calibrate to human labels, the authors show . |
Position: From Noise to Signal to Selbstzweck - Reframing Human Label Variation in the Era of Post-training in NLP (2026.findings-acl)
Copied to clipboard
| Challenge: | Human Label Variation (HLV) refers to legitimate disagreement in annotation . current preference-learning datasets routinely collapse multiple annotations into a single label . |
| Approach: | They propose to preserve human label variation as an embodiment of pluralism . they argue that disagreement in annotations should be treated as a selfzweck . |
| Outcome: | The proposed approach preserves pluralism and human pluralismos, the authors argue . they argue that disagreements in annotations should be treated as a selfzweck . |
LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evidence of human label variation in Natural Language Inference (NLI) however, within-label variation is an additional challenge. |
| Approach: | They propose a linguistically-informed taxonomy for categorizing free-text explanations in English that captures different reasoning strategies behind NLI explanations with a particular focus on within-label variation. |
| Outcome: | The proposed taxonomy can be used to classify explanations in English using a linguistically-informed taxonomies. |