| Challenge: | Using machines to correct factual errors is in high demand and requires a significant amount of human effort. |
| Approach: | They propose a zero-shot framework that asks questions about input claims and seeks correct answers from the given evidence to correct factual errors faithfully. |
| Outcome: | The proposed framework outperforms fully-supervised methods on the FEVER and SciFact datasets and is more faithful. |
Similar Papers
Zero-Shot Fact Verification via Natural Logic and Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advancements in fact verification systems with natural logic have enhanced their explainability by aligning claims with evidence through set-theoretic operators, providing faithful justifications. |
| Approach: | They propose a method that utilizes the generalization capabilities of instruction-tuned large language models to provide faithful justifications. |
| Outcome: | The proposed method outperforms other systems that were not specifically trained on natural logic data, and achieves an average accuracy improvement of 8.96 points over the baseline. |
Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains (2024.eacl-short)
Copied to clipboard
| Challenge: | Recent work has shown that large language models can generate zero-shot summaries without explicit supervision that are often comparable or even preferred to manually composed reference summary. |
| Approach: | They evaluate large language models (LLMs) that generate zero-shot summaries without explicit supervision that are often comparable to manual reference summary . they acquire annotations from domain experts to identify inconsistencies in summaires and categorize errors. |
| Outcome: | The proposed model outperforms fine-tuned models in biomedical articles and legal bills across specialized domains. |
Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work evaluates faithfulness using models trained on related tasks or in-domain synthetic data. |
| Approach: | They propose to do zero-shot faithfulness evaluation with a foundation language model. |
| Outcome: | The proposed model outperforms ChatGPT on faithfulness and inconsistency detection with 24x fewer parameters and is competitive with existing models. |
Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to optimism that they can serve as reliable evaluators of natural language outputs. |
| Approach: | They propose to use factuality metrics to evaluate natural language outputs . they find they misestimate the factual accuracy of NLG systems . |
| Outcome: | The proposed metrics are inconsistent with each other and often misestimate the factual accuracy of NLG systems, causing biases against paraphrased outputs and outputs that draw upon faraway parts of the source documents. |
TRUE: Re-evaluating Factual Consistency Evaluation (2022.naacl-main)
Copied to clipboard
Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, Yossi Matias
| Challenge: | Grounded text generation systems often generate factual inconsistencies, hindering their real-world applicability. |
| Approach: | They propose a method to assess factual consistency metrics on standardized texts . they recommend NLI and question generation-and-answering-based methods as starting points . |
| Outcome: | The proposed method is more actionable and interpretable than previous methods. |
Zero-shot Fact Verification by Claim Generation (2021.acl-short)
Copied to clipboard
| Challenge: | Existing methods for fact verification require large datasets, which can be expensive. |
| Approach: | They propose a framework for training a robust fact verification model by using automatically generated claims that can be supported, refuted, or unverifiable from evidence from Wikipedia. |
| Outcome: | The proposed framework reduces the demand for human-annotated training data and improves a model's F1 from 50% to 77%, equivalent in performance to 2K+ manually-curated examples. |
On The Ingredients of an Effective Zero-shot Semantic Parser (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have performed zero-shot learning by synthesizing training examples of canonical utterances and programs from a grammar, and further paraphrasing these utterrances to improve linguistic diversity. |
| Approach: | They propose to bridge gaps between canonical and real-world user-issued examples by using stronger paraphrasers and improved grammars. |
| Outcome: | The proposed model achieves strong performance on two semantic parsing benchmarks with zero labeled data. |
FLEEK: Factual Error Detection and Correction with Evidence Retrieved from External Knowledge (2023.emnlp-demo)
Copied to clipboard
Farima Fatahi Bayat, Kun Qian, Benjamin Han, Yisi Sang, Anton Belyy, Samira Khorshidi, Fei Wu, Ihab Ilyas, Yunyao Li
| Challenge: | Existing large language models (LLMs) have a tendency to hallucinate and provide creative and fluent responses that are not factually accurate. |
| Approach: | They propose a tool that automatically extracts factual claims from text, gathers evidence from external knowledge sources, evaluates the factuality of each claim, and suggests revisions for identified errors. |
| Outcome: | The proposed tool detects errors in text and evaluates their factuality and suggests revisions based on the collected evidence. |
LoReFact: Bridging the Logic Gap in Fact-Checking (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing fact-checking methods focus on verification of individual facts, overlooking logical dependencies . a recent study shows that text containing logical errors may still be misjudged as factual . |
| Approach: | They propose a content–logic coupled factuality evaluation paradigm that conceptualizes factual dimension along two complementary dimensions: content factualism and logic factuity. |
| Outcome: | The proposed paradigm bridges the gap between factual verification and content factuality . it incorporates the logical dimension and a logic-aware metric to expose and penalize logical fallacies. |
Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports (2020.acl-main)
Copied to clipboard
| Challenge: | Existing abstractive summarization models do not guarantee factual correctness of summaries . |
| Approach: | They propose a framework where models evaluate factual correctness by fact-checking it against its reference using an information extraction module. |
| Outcome: | The proposed method significantly improves the factual correctness and overall quality of outputs over a competitive neural summarization system, producing radiology summaries that approach the quality of human-authored ones. |