Can Unconfident LLM Annotations Be Used for Confident Conclusions? (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown high agreement with human raters across a variety of tasks, demonstrating potential to ease the challenges of human data collection. |
| Approach: | They propose a method that combines LLM annotations and LLM confidence indicators to strategically select which human annotations to use. |
| Outcome: | The proposed method produces accurate estimates and valid confidence intervals while reducing the number of human annotations by over 25%. |
Similar Papers
A Survey of Confidence Estimation and Calibration in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks in various domains, but they can be unreliable due to factual errors in their generations. |
| Approach: | They summarize recent advances in LLM confidence estimation and calibration and outline their main lessons learned. |
| Outcome: | The proposed methods can be used to assess the reliability of models and to calibrate them across tasks. |
All Roads Lead to Rome: Graph-Based Confidence Estimation for Large Language Model Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for confidence estimation are primarily designed for factual QA tasks and fail to generalize to reasoning tasks. |
| Approach: | They propose a set of training-free, graph-based confidence estimation methods tailored to reasoning tasks that exploit graph properties such as centrality, path convergence, and path weighting. |
| Outcome: | The proposed methods improve confidence estimation and performance on two downstream tasks. |
Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown impressive performance in many annotation tasks, including subjective tasks common in content moderation and text analysis in the social sciences. |
| Approach: | They propose to give crowdworkers LLM-generated annotation suggestions to "review" LLMs for subjective tasks can impact model performance and analysis downstream . |
| Outcome: | The proposed approach improves self-reported confidence in annotators and models . it also significantly improves model performance by analyzing human-approved datasets. |
Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized inference across diverse natural language tasks, with larger models performing better but at higher computational costs. |
| Approach: | They propose a confidence-driven strategy that dynamically selects the most suitable model based on confidence estimates. |
| Outcome: | The proposed approach reduces token usage by approximately 60% and improves cost efficiency on the Massive Multitask Language Understanding (MMLU) benchmark. |
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback (2023.emnlp-main)
Copied to clipboard
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, Christopher Manning
| Challenge: | Recent studies have shown that unsupervised pre-training produces large language models whose conditional probabilities are remarkably well-calibrated. |
| Approach: | They propose to use verbalized confidences to extract confidence from large language models with reinforcement learning from human feedback to improve their accuracy. |
| Outcome: | The proposed methods reduce the expected calibration error by 50% for RLHF-LMs such as ChatGPT, GPT-4, and Claude. |
Characterizing the Confidence of Large Language Model-Based Automatic Evaluation Metrics (2024.eacl-short)
Copied to clipboard
| Challenge: | Recent studies have focused on using Large Language Models (LLMs) to evaluate NLP tasks automatically. |
| Approach: | They characterize LLM evaluators’ confidence in ranking candidate NLP models and develop a configurable Monte Carlo simulation method to compensate for loss of correlation. |
| Outcome: | The proposed method can reach 95% confidence rankings of candidate models with reasonable evaluation set sizes. |
The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have shown capabilities close to human performance in various analytical tasks. |
| Approach: | They investigate the efficiency and accuracy of Large Language Models in specialized tasks . they integrate LLMs with expert annotators to observe the impact of LLM suggestions . |
| Outcome: | The proposed model improves task completion speed but introduces anchoring bias . the proposed model is not suitable for open-ended analysis, but is capable of handling specialized tasks. |
Evaluating Large Language Models for Confidence-based Check Set Selection (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models have shown promise in automating high-labor data tasks, but their tendency to answer despite uncertainty and their difficulty handling long input contexts robustly are key challenges for adoption. |
| Approach: | They propose to use LLMs to prioritize information needing human judgment to identify low-confidence outputs for human review through "check set selection" using social media monitoring, they define the "check sets" as a list of tweets escalated to the disaster manager when the LLM has the least confidence. |
| Outcome: | The proposed approach outperforms random-sample check set selection in disaster tweet classification. |
The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection (2025.findings-naacl)
Copied to clipboard
Tomáš Horych, Christoph Mandl, Terry Ruas, Andre Greiner-Petter, Bela Gipp, Akiko Aizawa, Timo Spinde
| Challenge: | Recent research suggests using Large Language Models (LLMs) to automate the annotation process, reducing these costs while maintaining data quality. |
| Approach: | They propose to use Large Language Models to automate annotation process and train classifiers on large datasets. |
| Outcome: | The proposed model outperforms all of the annotator LLMs on two media bias benchmark datasets (BABE and BASIL) while maintaining data quality. |
Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models’ Uncertainty? (2025.acl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used in high-stakes domains, but their confidence is inconsistent in out-of-distribution scenarios. |
| Approach: | They define "marker confidence" as the observed accuracy when a model employs an epistemic marker. |
| Outcome: | The proposed model generalizes well within the same distribution, but its confidence is inconsistent in out-of-distribution scenarios. |