Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers (2024.findings-emnlp)
Copied to clipboard
Yuxia Wang, Revanth Gangi Reddy, Zain Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, Preslav Nakov
| Challenge: | Large language models generate naturally sounding answers over a broad range of human inquiries, but they often generate answers that contradict real-world facts. |
| Approach: | They propose a framework for annotating and evaluating the factuality of large language models . they propose 'factcheck-bench' which provides a multi-stage annotation scheme . |
| Outcome: | The proposed framework outperforms several popular LLM fact-checkers in claim, sentence, and document levels. |
Similar Papers
OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs (2025.coling-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Hasan Iqbal, Georgi N. Georgiev, Jiahui Geng, Iryna Gurevych, Preslav Nakov
| Challenge: | Large language models (LLMs) generate naturallysounding answers over a broad range of human inquiries, but they still produce content that deviates from real-world facts. |
| Approach: | They propose a framework for building customized automatic fact-checking systems, benchmarking their accuracy, evaluating factuality of LLMs, and verifying claims in a document. |
| Outcome: | The proposed framework assesses the factuality of free-form responses in open domains and evaluates factually of LLMs. |
FactLens: Benchmarking Fine-Grained Fact Verification (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown impressive capability in language generation and understanding, but their tendency to hallucinate and produce factually incorrect information remains a key limitation. |
| Approach: | They propose a benchmark to evaluate fine-grained fact verification where claims are broken down into smaller sub-claims for individual verification. |
| Outcome: | The proposed model enables more precise identification of inaccuracies, improved transparency, and reduced ambiguity in evidence retrieval. |
OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Large language models (LLMs) often produce content that deviates from real-world facts. |
| Approach: | They developed a unified framework to assess the factuality of large language models . open-sourced framework is publicly available as a Python library and web service . |
| Outcome: | OpenFactCheck is open-sourced and publicly released as a Python library and web service. |
Generating Benchmarks for Factuality Evaluation of Language Models (2024.eacl-long)
Copied to clipboard
Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, Yoav Shoham
| Challenge: | Existing methods for factuality evaluation of LLM generation focus on facts sampled from the LM itself and might under-represent domain specific or rare facts. |
| Approach: | They propose a method that transforms a factual corpus into a benchmark evaluating an LM's propensity to generate true facts from the corpus . |
| Outcome: | The proposed framework transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements. |
MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current methods for fact-checking are based on verifying each piece of a model against potential evidence using an LLM. |
| Approach: | They propose a method that builds small fact-checking models that have GPT-4-level performance but 400x lower cost. |
| Outcome: | The proposed model outperforms other models and reaches GPT-4 accuracy. |
ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for evaluating factual consistency are primarily designed for short summaries of isolated code snippets. |
| Approach: | They propose a reference-free and fine-grained method for evaluating factual consistency in real-world code summaries. |
| Outcome: | The proposed method achieves highest correlation with human judgment among 13 baselines, improving 15-18% over the previous state-of-the-art. |
FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Language models (LMs) generate false or unverifiable content, often known as hallucination, despite ongoing efforts to enhance their factuality. |
| Approach: | They propose a tool that measures LMs’ factuality in real-world user interactions by evaluating their factual accuracy and categorizing content units as Supported, Unsupported, or Undecidable based on Web-retrieved evidence. |
| Outcome: | The proposed evaluation pipeline measures language models’ factuality in real-world user interactions. |
SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization (2023.emnlp-main)
Copied to clipboard
Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, Chien-Sheng Wu
| Challenge: | Existing factual consistency benchmarks are inadequate to detect factual inconsistencies in LLMs. |
| Approach: | They propose a protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. |
| Outcome: | The proposed method is 20 times more cost-effective per sample and highly reproducible, as it estimates inter-annotator agreement at about 0.9. |
X-Fact: A New Benchmark Dataset for Multilingual Fact Checking (2021.acl-short)
Copied to clipboard
| Challenge: | Several fact-checking initiatives, such as PolitiFact, expend manual labor to investigate and determine the truthfulness of viral statements. |
| Approach: | They propose a multilingual dataset for factual verification of naturally existing claims . they use a benchmark to evaluate the multilingual models . |
| Outcome: | The proposed model achieves an F-score of around 40%, suggesting it is a challenging benchmark for multilingual fact-checking models. |
Self-Checker: Plug-and-Play Modules for Fact-Checking with Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for fact-checking text generated by large language models are expensive and time-consuming. |
| Approach: | They propose a plug-and-play framework that harnesses large language models for efficient fact-checking in a few-shot manner. |
| Outcome: | The proposed framework is compared with state-of-the-art models and shows that it can be used to speed up fact-checking in a few-shot manner. |