Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown impressive performance in many annotation tasks, including subjective tasks common in content moderation and text analysis in the social sciences. |
| Approach: | They propose to give crowdworkers LLM-generated annotation suggestions to "review" LLMs for subjective tasks can impact model performance and analysis downstream . |
| Outcome: | The proposed approach improves self-reported confidence in annotators and models . it also significantly improves model performance by analyzing human-approved datasets. |
Similar Papers
The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have shown capabilities close to human performance in various analytical tasks. |
| Approach: | They investigate the efficiency and accuracy of Large Language Models in specialized tasks . they integrate LLMs with expert annotators to observe the impact of LLM suggestions . |
| Outcome: | The proposed model improves task completion speed but introduces anchoring bias . the proposed model is not suitable for open-ended analysis, but is capable of handling specialized tasks. |
Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals’ Subjective Text Perceptions (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that LLMs perform poorly when prompted with sociodemographic attributes, suggesting limited inherent sociodemography knowledge. |
| Approach: | They propose to train large language models to be accurate sociodemographic models of annotator variation by using a curated dataset of five tasks with standardized sociodemography. |
| Outcome: | The proposed models improve in sociodemographic prompting when trained but this performance gain is largely due to models learning annotator-specific behaviour rather than sociodemography. |
Task Assignment meets Annotator Modeling: Human-LLM Collaborative Annotation with Constraints (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing approaches to label annotation are labor-intensive and time-consuming. |
| Approach: | They propose a framework that estimates per-task accuracy from task features using a learning from crowds model and incorporates these estimations into a linear programming formulation that assigns tasks under practical constraints. |
| Outcome: | The proposed method achieves comparable accuracy to baseline methods while satisfying given constraints. |
MEGAnno+: A Human-LLM Collaborative Annotation System (2024.eacl-demo)
Copied to clipboard
| Challenge: | Large language models (LLMs) can label data faster and cheaper than humans . however, they may fall short in understanding of complex contexts, leading to incorrect labels . |
| Approach: | They propose a collaborative approach where humans and LLMs work together to produce reliable labels. |
| Outcome: | The proposed system can produce reliable and high-quality labels faster and cheaper than humans . compared to traditional models, it can generate labels faster, at a lower cost . |
Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) offer new opportunities to enhance the annotation process, particularly for detecting label errors in existing datasets. |
| Approach: | They propose to use an ensemble of large language models to flag mislabeled examples by using an LLM-as-a-judge approach to detect label errors in existing datasets. |
| Outcome: | The proposed method improves label accuracy and consistency in large language models. |
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2025.acl-short)
Copied to clipboard
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
| Challenge: | Existing evaluations of NLP models with LLMs are based on human judgments . however, there are concerns about their validity and reproducibility in proprietary models . |
| Approach: | They evaluate 11 current LLMs for their ability to replicate annotations. they show substantial variance across models and datasets. |
| Outcome: | The proposed model can replicate human annotations on 20 NLP datasets and show substantial variance across models and datasets. |
Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown considerable promise for annotation purposes, but questions remain about their ability to capture human label variation (HLV) label variation is genuine disagreement between annotators observed across NLP tasks. |
| Approach: | They investigate how label and explanation variation manifests within and across LLMs with respect to the Natural Language Inference task. |
| Outcome: | The proposed models generate label distributions similar to humans but exhibit distinct, idiosyncratic judgments and disagreement patterns. |
Can Unconfident LLM Annotations Be Used for Confident Conclusions? (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown high agreement with human raters across a variety of tasks, demonstrating potential to ease the challenges of human data collection. |
| Approach: | They propose a method that combines LLM annotations and LLM confidence indicators to strategically select which human annotations to use. |
| Outcome: | The proposed method produces accurate estimates and valid confidence intervals while reducing the number of human annotations by over 25%. |
A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent research has shown that explanations provide valuable information for understanding human label variation (HLV) Large language models (LLMs) can approximate HJD from a few human-provided label-explanation pairs, but collecting explanations for every label is still time-consuming. |
| Approach: | They propose to use Large Language Models (LLMs) as annotators to generate model explanations for a few given human labels. |
| Outcome: | The proposed models can generate human-provided explanations from human labels, but they are still time-consuming. |
The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are widely used in NLP research but there is no standard or rigorous procedure to determine whether they can replace humans. |
| Approach: | They propose a statistical procedure that requires only a modest subset of annotated examples to justify using LLM annotations. |
| Outcome: | The proposed procedure compares LLMs with open-source LLM annotators and judges on ten language and vision-language tasks. |