Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Despite the progress of factual evaluation methods, they are limited in their opacity and lack the ability to assess the factuality of the summaries. |
| Approach: | They propose to use a meta-evaluation methodology to diagnose the fine-grained strengths and weaknesses of 6 existing top-performing metrics over 24 diagnostic test datasets. |
| Outcome: | The proposed method diagnoses the strengths and weaknesses of 6 existing top-performing metrics over 24 diagnostic test datasets and searches for directions for further improvement by data augmentation. |
Similar Papers
Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors (2023.acl-long)
Copied to clipboard
Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, Greg Durrett
| Challenge: | Abstractive summarization systems still include factual errors in generated summaries despite recent improvements in factuality detection . |
| Approach: | They aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. |
| Outcome: | The proposed method improves on the ChatGPT-based model and shows that it is not superior for all error types. |
Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to optimism that they can serve as reliable evaluators of natural language outputs. |
| Approach: | They propose to use factuality metrics to evaluate natural language outputs . they find they misestimate the factual accuracy of NLG systems . |
| Outcome: | The proposed metrics are inconsistent with each other and often misestimate the factual accuracy of NLG systems, causing biases against paraphrased outputs and outputs that draw upon faraway parts of the source documents. |
Gradient-Based Adversarial Factual Consistency Evaluation for Abstractive Summarization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization models often produce inconsistent statements or false facts. |
| Approach: | They propose an efficient weak-supervised adversarial data augmentation approach to generate factual consistency datasets by backpropagating gradients on token embeddings. |
| Outcome: | The proposed model can make interpretable factual errors tracing on public datasets and is cost-effective. |
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics (2021.naacl-main)
Copied to clipboard
| Challenge: | Modern summarization models generate fluent but often factually unreliable outputs. |
| Approach: | They propose to use human annotations to identify different categories of factual errors and benchmark factuality metrics to improve summarization evaluation. |
| Outcome: | The proposed method identifies the proportion of different categories of factual errors and benchmarks their human judgements as well as their specific strengths and weaknesses. |
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in text generation systems produce fluent, coherent, relevant, and factually correct text. |
| Approach: | They propose a metaevaluation framework for evaluating factuality evaluation metrics . they propose five necessary conditions to evaluate factual metrics on diagnostic factuity data . |
| Outcome: | The proposed framework provides robust evaluation that is extensible to multiple types of factual consistency and standard generation metrics, including QA metrics. |
Annotating and Modeling Fine-grained Factuality in Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent abstractive summarization systems produce factual errors that are not faithful to the input . current methods are lacking in identifying what errors are most important to target . |
| Approach: | They use synthetic and human-labeled data to identify factual errors in summarization and train models on the factuality detection task. |
| Outcome: | The proposed model detects factual errors on word, dependency, and sentence levels. |
Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports (2020.acl-main)
Copied to clipboard
| Challenge: | Existing abstractive summarization models do not guarantee factual correctness of summaries . |
| Approach: | They propose a framework where models evaluate factual correctness by fact-checking it against its reference using an information extraction module. |
| Outcome: | The proposed method significantly improves the factual correctness and overall quality of outputs over a competitive neural summarization system, producing radiology summaries that approach the quality of human-authored ones. |
A Critical Look at Meta-evaluating Summarisation Evaluation Metrics (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Effective summarisation evaluation metrics enable researchers and practitioners to compare different summarization systems efficiently. |
| Approach: | They argue that evaluation metrics are primarily meta-evaluated on news summarisation datasets and that there has been a noticeable shift in research focus towards evaluating the faithfulness of generated summaries. |
| Outcome: | The evaluation metrics are primarily meta-evaluated on news summarisation datasets and there has been a noticeable shift in research focus towards evaluating the faithfulness of generated summaries. |
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization models often generate inconsistent summaries containing factual errors or fabricated content. |
| Approach: | They propose to generate representative examples of non-factual summaries through infilling language models and train a robust fact-correction model to post-edit them to improve factual consistency. |
| Outcome: | The proposed model outperforms previous methods in correcting factual errors on two popular summarization datasets. |
Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization systems have a lack of a defined definition for the task . factual consistency is a key factor in summarizing, but there are still deficiencies . a new study shows that summarized summarisation models achieve improved performance . |
| Approach: | They propose a filtered summarization dataset with improved factual consistency to address this problem . they argue that the dataset should become a valid benchmark for developing and evaluating summarizing systems . |
| Outcome: | The proposed model improves on a popular summarization dataset with improved factual consistency. |