LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing metrics for text summarisation have restrictive token limits, limiting their effectiveness. |
| Approach: | They propose a human-annotated data set for evaluating automatic factuality metrics . they propose 'longDocFACTScore' framework which can be extended to any length document . |
| Outcome: | The proposed framework outperforms state-of-the-art metrics in evaluating long document summarisation data sets. |
Similar Papers
How Far are We from Robust Long Abstractive Summarization? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization has made tremendous progress in recent years . however, even under a short document setting, abstractive models often generate summaries that are repetitive, ungrammatical, and factually inconsistent with the source. |
| Approach: | They perform fine-grained human annotations to evaluate long document abstractive summarization systems and develop factual consistency metrics. |
| Outcome: | The proposed model can generate more relevant summaries but not factual ones. |
Stress Testing Factual Consistency Metrics for Long-Document Summarization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing short-form summarization metrics struggle with input length limitations and long-range dependencies. |
| Approach: | They propose to evaluate the reliability of six widely used reference-free factuality metrics in the long-document setting by applying seven factually-preserving perturbations to summaries. |
| Outcome: | The proposed short-form summarization metrics struggle with long-range dependencies and input length limitations. |
A Tale of Evaluating Factual Consistency: Case Study on Long Document Summarization Evaluation (2025.findings-acl)
Copied to clipboard
| Challenge: | Despite the recent progress for summarization models in producing fluent summaries, they still encounter challenges when long sequences of generated texts and inputs (over thousands of words) need to be evaluated. |
| Approach: | They conduct a systematic analysis of factual-consistency evaluation systems across four long-document datasets and examine the relationship between sentence-level and summary-level model performance. |
| Outcome: | The proposed models can achieve higher recall in error detection for older summaries, yet struggle with false positives and fine-grained error detection. |
Evaluating the Factual Consistency of Abstractive Text Summarization (2020.emnlp-main)
Copied to clipboard
| Challenge: | a weakly-supervised approach is needed to verify factual consistency . auxiliary span extraction tasks are useful for verifying factual consistent summaries . |
| Approach: | They propose a weakly-supervised approach for verifying factual consistency . they transfer the model to summaries generated by several neural models . |
| Outcome: | The proposed approach outperforms models trained with strong supervision on source documents and human evaluations. |
Evaluating and Improving Factuality in Multimodal Abstractive Summarization (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current factuality metrics do not account for vision modality, thus are not adequate for vision-and-language summarization. |
| Approach: | They propose a weighted combination of CLIPScore and BERTScore to evaluate factuality for abstractive document summarization. |
| Outcome: | The proposed metric outperforms existing factuality metrics on four factuity metric-evaluation benchmarks and is robust to human judgments. |
VeriScore: Evaluating the factuality of verifiable claims in long-form text generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing metrics for evaluating the factuality of long-form text assume that every claim is verifiable. |
| Approach: | They propose a metric to evaluate factuality in diverse long-form generation tasks . they use open-weight language models to extract verifiable and unverifiably content . |
| Outcome: | The proposed metric can be implemented with either closed or fine-tuned open-weight language models. |
Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate factual consistency in text summarization neglect the intrinsic cause of factual inconsistency or rely on auxiliary tasks. |
| Approach: | They propose a method to evaluate the factual consistency in text summarization via counterfactual estimation, which formulates the causal relationship between source document, generated summary, and the language prior. |
| Outcome: | The proposed metric improves correlation with human judgments and convenience of usage on three public abstractive text summarization datasets. |
Fact-based Content Weighting for Evaluating Abstractive Summarisation (2020.acl-main)
Copied to clipboard
| Challenge: | Abstractive summarisation is notoriously hard to evaluate since word-overlap-based metrics are insufficient. |
| Approach: | They propose a new evaluation metric which is based on fact-level content weighting, relating the facts of the document to the facts in the summary. |
| Outcome: | The proposed evaluation metric is highly correlated to human perception and compares favourably to the recent manual highlight-based metric of Hardy et al. |
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization (2023.eacl-main)
Copied to clipboard
| Challenge: | Human evaluation is labor-intensive, expensive to scale, and difficult to design. |
| Approach: | They propose a set of guidelines for human evaluation of faithfulness in long-form summaries that address the following challenges: (1) How can we achieve high inter-annotator agreement on faithfulness scores? (2) How can our annotator minimize workload while maintaining accurate faithfulness? |
| Outcome: | The proposed framework reduces inter-annotator variance in faithfulness scores while minimizing annotator workload while maintaining accuracy. |
Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing summarization systems can generate fluent summaries, but their ability to produce factually consistent summary remains questionable. |
| Approach: | They propose a framework that decomposes long texts into discourse-inspired chunks and utilizes discourse information to better aggregate sentence-level scores predicted by NLI models. |
| Outcome: | The proposed framework shows better performance over multiple benchmarks, focusing on long document summarization. |