Papers with confidence
The Role of Context and Uncertainty in Shallow Discourse Parsing (2022.coling-1)
Copied to clipboard
| Challenge: | Discourse parsing has proven to be useful for a number of NLP tasks that require complex reasoning. |
| Approach: | They hypothesize that context plays an important role in accurate human annotation and add uncertainty measures can improve model accuracy and calibration. |
| Outcome: | The proposed model can be better calibrated by adding uncertainty measures to models with better accuracy and calibration. |
A Close Look into the Calibration of Pre-trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) may fail in giving reliable estimates of their predictive uncertainty. |
| Approach: | They conduct fine-grained control experiments to study the dynamic change in PLMs’ calibration performance in training. |
| Outcome: | The proposed methods significantly reduce PLMs’ confidence in wrong predictions. |
When to Trust LLMs: Aligning Confidence with Response Quality (2024.findings-acl)
Copied to clipboard
Shuchang Tao, Liuyi Yao, Hanxing Ding, Yuexiang Xie, Qi Cao, Fei Sun, Jinyang Gao, Huawei Shen, Bolin Ding
| Challenge: | Existing methods express reliability by confidence level, but lack objective guidance . Existing approaches express reliability but lack guidance on when to trust LLMs . |
| Approach: | They propose a reward-based approach to align confidence with quality to ensure reliability . they propose 'conqORD' to help model to verbalize greater confidence for higher quality responses . |
| Outcome: | Experiments show that CONQORD significantly improves confidence and response accuracy . the proposed approach can be used to determine reliability of large language models . |
Zero-Shot Data Maps. Efficient Dataset Cartography Without Model Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to diagnose large annotated datasets require the fitting of a strong model to the dataset. |
| Approach: | They propose a new approach to compute confidence and variability over an ensemble of zero-shot models constructed with different but semantically equivalent label descriptions. |
| Outcome: | The proposed method can be used to diagnose large annotated datasets with accuracy up to 14x faster than the current method. |
BAGELS: Benchmarking the Automated Generation and Extraction of Limitations from Scholarly Text (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a growing number of scientific publications have limitations as a source of uncertainty. |
| Approach: | They propose a computational architecture for extracting and generating limitations from scholarly papers using a novel Retrieval Augmented Generation technique. |
| Outcome: | The proposed architecture extracts limitations from ACL, NeurIPS, and PeerJ papers and supplementes them with external reviews. |
Reflections & Resonance: Two-Agent Partnership for Advancing LLM-based Story Annotation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for story annotation require a meticulous and resourceintensive effort, but the advent of advanced computational tools like GPT-4 can streamline the process and mitigate common limitations. |
| Approach: | They propose a multi-agent system that generates tailored prompts for a large language model and provides feedback to refine the initial prompts. |
| Outcome: | The proposed system significantly improves the model's reconstruction accuracy and confidence, demonstrating that dynamic interaction between agents significantly boosts the annotation process's precision and efficiency. |
What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Curriculum learning (CL) orders data corpus by difficulty, but prior work employs disparate difficulty metrics and training setups. |
| Approach: | They propose a framework that decomposes curriculum difficulty into five dimensions: Problem Difficulty, Model Surprisal, Confidence Margin, Predictive Uncertainty and Decision Variability. |
| Outcome: | The proposed framework decomposes curriculum difficulty into five dimensions . the results show that no curriculum strategy dominates universally . |