| Challenge: | Annotations with incorrect label or boundaries count as two errors instead of one, despite being closer to the target annotation than false positives or false negatives. |
| Approach: | They propose an algorithm for error identification in flat and multi-level annotations and propose a procedure for calculating meaningful precision, recall, and F1-scores based on the more fine-grained error types. |
| Outcome: | The proposed procedure prevents double penalties and allows for a more detailed error analysis, providing more insight into the actual weaknesses of a system. |
Similar Papers
FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Existing tools to evaluate long text outputs are lacking in the field of NLP . human rating and error analysis remains a crucial component for any evaluation of long text generation. |
| Approach: | They propose a web-based toolkit to collect fine-grained error annotations for long texts . they use a taxonomy to identify errors and assign them to text spans . |
| Outcome: | The proposed tool can be used to evaluate the coherence of long generated summaries. |
Don’t waste a single annotation: improving single-label classifiers through soft labels (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for annotating data are limited by ambiguity and lack of context in data samples. |
| Approach: | They challenge the traditional approach of annotating data by only providing a single label for each sample and annotator disagreement is discarded . instead, they use additional annotation information such as confidence, secondary label and disagreement to generate soft labels. |
| Outcome: | The proposed method improves model performance and calibration on the hard label test set. |
Annotating and Modeling Fine-grained Factuality in Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent abstractive summarization systems produce factual errors that are not faithful to the input . current methods are lacking in identifying what errors are most important to target . |
| Approach: | They use synthetic and human-labeled data to identify factual errors in summarization and train models on the factuality detection task. |
| Outcome: | The proposed model detects factual errors on word, dependency, and sentence levels. |
An Evaluation Resource for Grounding Translation Errors (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current fine-grained error analyses do not ground the errors to the reasons why the annotated text spans are erroneous. |
| Approach: | They use a bi-directional grounding scheme to ground erroneous text in two directions . if the error spans of both directions are consistent, the explanation is valid . |
| Outcome: | The proposed grounding process improves translation error detection significantly. |
Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization (2023.acl-long)
Copied to clipboard
| Challenge: | Existing work on factual inconsistency in abstractive summarization addresses this problem. |
| Approach: | They propose a dataset with fine-grained factual error annotations named DIASUMFACT and an unsupervised model named ENDERANKER. |
| Outcome: | The proposed model performs on par with the state-of-the-art models while requiring fewer resources. |
Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) offer new opportunities to enhance the annotation process, particularly for detecting label errors in existing datasets. |
| Approach: | They propose to use an ensemble of large language models to flag mislabeled examples by using an LLM-as-a-judge approach to detect label errors in existing datasets. |
| Outcome: | The proposed method improves label accuracy and consistency in large language models. |
Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors (2023.acl-long)
Copied to clipboard
Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, Greg Durrett
| Challenge: | Abstractive summarization systems still include factual errors in generated summaries despite recent improvements in factuality detection . |
| Approach: | They aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. |
| Outcome: | The proposed method improves on the ChatGPT-based model and shows that it is not superior for all error types. |
Fine-grained Fallacy Detection with Human Label Variation (2025.naacl-long)
Copied to clipboard
| Challenge: | Fallacy detection is an open challenge in NLP and has shown to be intrinsically difficult for both humans and machines. |
| Approach: | They propose a framework that minimizes annotation errors whilst keeping signals of human label variation. |
| Outcome: | The proposed framework minimizes annotation errors while keeping signals of human label variation. |
MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types (2023.findings-acl)
Copied to clipboard
Keerthiram Murugesan, Sarathkrishna Swaminathan, Soham Dan, Subhajit Chaudhury, Chulaka Gunasekara, Maxwell Crouse, Diwakar Mahajan, Ibrahim Abdelaziz, Achille Fokoue, Pavan Kapanipathi, Salim Roukos, Alexander Gray
| Challenge: | Existing evaluation metrics for machine text are inadequate to capture quality of text . a recent study has focused on task-specific evaluation metrics or on properties of machine-generated text based on mismatch errors . |
| Approach: | They propose a new evaluation scheme based on fine-grained mismatch errors . they propose 13 mismatch error types to guide the model for better prediction of human judgments . |
| Outcome: | The proposed evaluation scheme is based on mismatch errors in 7 NLP tasks . the mismatch error types guide the model for better prediction of human judgments . |
Coarse2Fine: Fine-grained Text Classification on Coarsely-grained Annotated Data (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing text classification methods focus on a fixed label set, but many real-world applications require extending to new fine-grained classes as the number of samples per label increases. |
| Approach: | They propose a problem called coarse-to-fine grained classification that leverages label surface names as the only human guidance. |
| Outcome: | The proposed method outperforms existing methods on two real-world datasets. |