FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Language models (LMs) generate false or unverifiable content, often known as hallucination, despite ongoing efforts to enhance their factuality. |
| Approach: | They propose a tool that measures LMs’ factuality in real-world user interactions by evaluating their factual accuracy and categorizing content units as Supported, Unsupported, or Undecidable based on Web-retrieved evidence. |
| Outcome: | The proposed evaluation pipeline measures language models’ factuality in real-world user interactions. |
Similar Papers
FactEval: Evaluating the Robustness of Fact Verification Systems in the Era of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have made significant advances in every natural language processing task, but they are vulnerable to small perturbations in the inputs, raising concerns about their robustness in the real world. |
| Approach: | They propose a large-scale benchmark for extensive evaluation of LLMs in the fact verification domain covering 17 realistic word-level and character-level perturbations and 4 types of subpopulations. |
| Outcome: | The proposed model is brittle to small input changes and exhibits performance variations across different subpopulations. |
Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers (2024.findings-emnlp)
Copied to clipboard
Yuxia Wang, Revanth Gangi Reddy, Zain Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, Preslav Nakov
| Challenge: | Large language models generate naturally sounding answers over a broad range of human inquiries, but they often generate answers that contradict real-world facts. |
| Approach: | They propose a framework for annotating and evaluating the factuality of large language models . they propose 'factcheck-bench' which provides a multi-stage annotation scheme . |
| Outcome: | The proposed framework outperforms several popular LLM fact-checkers in claim, sentence, and document levels. |
Factuality of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Das, Preslav Nakov
| Challenge: | Large language models (LLMs) are factually incorrect, which limits their applicability in real-world scenarios. |
| Approach: | They analyze existing work to identify major challenges and their associated causes . they propose to evaluate LLMs using a variety of measures to mitigate factual errors . |
| Outcome: | The proposed methods are based on a variety of datasets and proposed strategies to mitigate factual errors. |
VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work focuses on accuracy and precision, but factuality evaluation is difficult due to inter-sentence dependencies. |
| Approach: | They introduce a factuality evaluation framework to enhance fact extraction . they also introduce 'factRBench' that evaluates both precision and recall . |
| Outcome: | The proposed framework enhances fact extraction by identifying incomplete and missing facts . it also evaluates precision and recall in long-form models, whereas prior work focuses on precision. |
FaStFact: Faster, Stronger Long-Form Factuality Evaluations in LLMs (2025.findings-emnlp)
Copied to clipboard
Yingjia Wan, Haochen Tan, Xiao Zhu, Xinyu Zhou, Zhiwei Li, Qingsong Lv, Changxuan Sun, Jiaqi Zeng, Yi Xu, Jianqiao Lu, Yinhong Liu, Zhijiang Guo
| Challenge: | Prior evaluation pipelines fail to evaluate factuality of long-form LLMs due to inefficiency and costly human assessment. |
| Approach: | They propose a fast and strong evaluation pipeline that can evaluate factuality of long-form LLMs . they propose 'faStFact' to reduce cost of web searching and inference calling . |
| Outcome: | The proposed evaluation pipeline achieves highest alignment with human evaluation and efficiency among existing baselines. |
FactLens: Benchmarking Fine-Grained Fact Verification (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown impressive capability in language generation and understanding, but their tendency to hallucinate and produce factually incorrect information remains a key limitation. |
| Approach: | They propose a benchmark to evaluate fine-grained fact verification where claims are broken down into smaller sub-claims for individual verification. |
| Outcome: | The proposed model enables more precise identification of inaccuracies, improved transparency, and reduced ambiguity in evidence retrieval. |
TRUEBench: Can LLM Response Meet Real-world Constraints as Productivity Assistant? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks fail to evaluate large language models' instruction-following capabilities . current benchmarks lack multilinguality, implicit constraints and multi-turn dialogue . |
| Approach: | a new benchmark is designed to evaluate large language models' instruction-following capabilities . the benchmark features input prompts across 12 languages and includes inter-instance multilingual instructions . |
| Outcome: | a new benchmark for large language models (LLMs) is designed to assess their performance in real-world settings. |
Teaching Language Models to Check Grounded Claim Factuality with Human Test-Taking Strategies (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for factuality checking require dataset-specific threshold tuning, while LLM-based approaches often use direct prompting. |
| Approach: | They propose to use a reading comprehension task to check for true/false claim factuality and prompt LLMs with explicit test-taking strategies for efficient reasoning. |
| Outcome: | The proposed method reduces token usage by over 80% compared to unguided open-ended reasoning and achieves competitive performance to more expensive alternatives. |
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models (2024.acl-long)
Copied to clipboard
Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral
| Challenge: | Existing work investigating the logical reasoning ability of large language models has focused only on a couple of inference rules of propositional and first-order logics. |
| Approach: | They propose to use a natural language question-answering dataset to evaluate the logical reasoning ability of large language models. |
| Outcome: | The proposed model performs poorly on a range of natural language questions using chain-of-thought prompting. |
Language Models Hallucinate, but May Excel at Fact Verification (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have produced non-factual outputs . however, current LLMs suffer from the hallucination issue . |
| Approach: | They propose to use instruction-tuned LLMs to generate factual outputs . they find that FLAN-T5-11B performs best as a fact verifier . |
| Outcome: | The proposed method outperforms more capable LLMs like GPT3.5 and ChatGPT in the human evaluation. |