| Challenge: | Large Language Models (LLMs) have shown impressive capability in language generation and understanding, but their tendency to hallucinate and produce factually incorrect information remains a key limitation. |
| Approach: | They propose a benchmark to evaluate fine-grained fact verification where claims are broken down into smaller sub-claims for individual verification. |
| Outcome: | The proposed model enables more precise identification of inaccuracies, improved transparency, and reduced ambiguity in evidence retrieval. |
Similar Papers
Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers (2024.findings-emnlp)
Copied to clipboard
Yuxia Wang, Revanth Gangi Reddy, Zain Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, Preslav Nakov
| Challenge: | Large language models generate naturally sounding answers over a broad range of human inquiries, but they often generate answers that contradict real-world facts. |
| Approach: | They propose a framework for annotating and evaluating the factuality of large language models . they propose 'factcheck-bench' which provides a multi-stage annotation scheme . |
| Outcome: | The proposed framework outperforms several popular LLM fact-checkers in claim, sentence, and document levels. |
Learning to Verify Summary Facts with Fine-Grained LLM Feedback (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have significantly enhanced the text summarization performance, but hallucination issues still occur in summaries. |
| Approach: | They propose a large-scale dataset containing fine-grained factual feedback on summaries that can be fine tuned by using Large Language Models (LLMs) they employ 10 distinct LLMs for diverse summary generation and Llama-3-70B-Instruct for feedback. |
| Outcome: | The proposed model outperforms models trained on smaller human-annotated datasets while maintaining high performance. |
FactEval: Evaluating the Robustness of Fact Verification Systems in the Era of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have made significant advances in every natural language processing task, but they are vulnerable to small perturbations in the inputs, raising concerns about their robustness in the real world. |
| Approach: | They propose a large-scale benchmark for extensive evaluation of LLMs in the fact verification domain covering 17 realistic word-level and character-level perturbations and 4 types of subpopulations. |
| Outcome: | The proposed model is brittle to small input changes and exhibits performance variations across different subpopulations. |
FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Language models (LMs) generate false or unverifiable content, often known as hallucination, despite ongoing efforts to enhance their factuality. |
| Approach: | They propose a tool that measures LMs’ factuality in real-world user interactions by evaluating their factual accuracy and categorizing content units as Supported, Unsupported, or Undecidable based on Web-retrieved evidence. |
| Outcome: | The proposed evaluation pipeline measures language models’ factuality in real-world user interactions. |
Claim Verification in the Age of Large Language Models: A Survey (2026.acl-srw)
Copied to clipboard
| Challenge: | Recent election cycles have seen a large number of false information spread across social media and news platforms. |
| Approach: | They propose a framework for automated claim verification using Large Language Models and Retrieval Augmented Generation. |
| Outcome: | The proposed frameworks are based on large-scale models and new methods such as Retrieval Augmented Generation (RAG). |
ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for evaluating factual consistency are primarily designed for short summaries of isolated code snippets. |
| Approach: | They propose a reference-free and fine-grained method for evaluating factual consistency in real-world code summaries. |
| Outcome: | The proposed method achieves highest correlation with human judgment among 13 baselines, improving 15-18% over the previous state-of-the-art. |
Towards Effective Extraction and Evaluation of Factual Claims (2025.acl-long)
Copied to clipboard
| Challenge: | Lack of a standardized evaluation framework impedes assessment and comparison of claim extraction methods. |
| Approach: | They propose a framework for evaluating claim extraction in the context of fact-checking . they also introduce Claimify, an LLM-based claim extraction method . |
| Outcome: | The proposed evaluation framework outperforms existing methods in the evaluation of claim extraction methods. |
Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification (2024.findings-acl)
Copied to clipboard
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov
| Challenge: | Large language models are notorious for producing erroneous claims in their output. |
| Approach: | They propose a fact-checking and hallucination detection pipeline based on token-level uncertainty quantification that removes the impact of uncertainty about what claim to generate on the current step and what surface form to use. |
| Outcome: | The proposed method can fact-check the atomic claims in the output of large language models. |
SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization (2023.emnlp-main)
Copied to clipboard
Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, Chien-Sheng Wu
| Challenge: | Existing factual consistency benchmarks are inadequate to detect factual inconsistencies in LLMs. |
| Approach: | They propose a protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. |
| Outcome: | The proposed method is 20 times more cost-effective per sample and highly reproducible, as it estimates inter-annotator agreement at about 0.9. |
Improving LLM Generations via Fine-Grained Self-Endorsement (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have demonstrated remarkable capabilities but can still fail frequently on knowledge-intensive tasks. |
| Approach: | They propose a self-endorsement framework that leverages fine-grained fact-level comparisons across multiple sampled responses. |
| Outcome: | The proposed framework can improve factuality of generations with simple prompts across scales of LLMs. |