WikiFactDiff: A Large, Realistic, and Temporally Adaptable Dataset for Atomic Factual Knowledge Update in Causal Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Factual update is a task of inserting, replacing, or removing facts in large language models. |
| Approach: | They present a dataset that describes the evolution of factual knowledge between two dates as a collection of simple facts divided into three categories: new, obsolete, and static. |
| Outcome: | The proposed dataset compares the state of the Wikidata knowledge base at 4 January 2021 and 27 February 2023. |
Similar Papers
StructFact: Reasoning Factual Knowledge from Structured Data with Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have made significant strides in natural language processing by leveraging their ability to comprehend and reason with factual knowledge. |
| Approach: | They propose a benchmark to evaluate the ability of large language models to reason with structured data for knowledge-intensive tasks. |
| Outcome: | Extensive tests on 10 common LLMs show that they struggle with heterogeneity of structured data during reasoning. |
FactCorrector: A Graph-Inspired Approach to Long-Form Factuality Correction of Large Language Models (2026.acl-long)
Copied to clipboard
Javier Carnerero-Cano, Massimiliano Pronesti, Radu Marinescu, Tigran T. Tchrakian, James Barry, Jasmina Gajcin, Yufang Hou, Alessandra Pascale, Elizabeth M. Daly
| Challenge: | Large language models (LLMs) often produce factually incorrect responses. |
| Approach: | They propose a new method that adapts across domains without retraining and leverages structured feedback to generate a correction. |
| Outcome: | The proposed method outperforms baseline methods on a VELI5 dataset and several popular long-form factuality datasets. |
When Facts Change: Temporal Knowledge Conflict Resolution in LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models are increasingly used in retrieval-augmented generation systems to reconcile knowledge conflicts between parametric memory and contextual inputs. |
| Approach: | They propose to use mutability to resolve temporal misalignment in large language models to compare stable and recently updated facts from Wikidata to determine if mutable models can serve as a mediating signal in this process. |
| Outcome: | The proposed model can produce reasoning for facts that actually changed but rarely for stable ones, whereas smaller models rarely detect conflict, while larger models detect it but fail to act on mutability judgments. |
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are impressive in solving tasks, but they can quickly be outdated after deployment. |
| Approach: | They provide a review of recent advances in aligning deployed large language models with the ever-changing world knowledge. |
| Outcome: | The proposed models can be used to perform various tasks directly through in-context learning or for further fine-tuning for domain-specific uses. |
Temporally Consistent Factuality Probing for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are used as an alternative knowledge base for many tasks. |
| Approach: | They propose a temporally consistent factuality probe task that extends the consistency probe in the temporal dimension. |
| Outcome: | The proposed task extends the definitions of existing metrics to represent consistent factuality across temporal dimension. |
When Benchmarks Age: Temporal Misalignment through Large Language Model Factuality Evaluation (2026.eacl-short)
Copied to clipboard
| Challenge: | Existing studies on LLM factuality evaluation have not investigated the reliability of static evaluation benchmarks. |
| Approach: | They examine five popular factuality benchmarks and eight LLMs released over different years to assess their reliability. |
| Outcome: | The proposed method compared five popular factuality benchmarks and eight LLMs released over different years. |
OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Large language models (LLMs) often produce content that deviates from real-world facts. |
| Approach: | They developed a unified framework to assess the factuality of large language models . open-sourced framework is publicly available as a Python library and web service . |
| Outcome: | OpenFactCheck is open-sourced and publicly released as a Python library and web service. |
Factuality of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Das, Preslav Nakov
| Challenge: | Large language models (LLMs) are factually incorrect, which limits their applicability in real-world scenarios. |
| Approach: | They analyze existing work to identify major challenges and their associated causes . they propose to evaluate LLMs using a variety of measures to mitigate factual errors . |
| Outcome: | The proposed methods are based on a variety of datasets and proposed strategies to mitigate factual errors. |
WikiAtomicEdits: A Multilingual Corpus of Wikipedia Edits for Modeling Language and Discourse (D18-1)
Copied to clipboard
| Challenge: | a corpus of 43 million atomic edits is available for Wikipedia edit history . edits are instances in which a human editor has inserted a single contiguous phrase into, or deleted a contigous phrase from, an existing sentence. |
| Approach: | They use Wikipedia edit history to mine atomic edits across 8 languages . they find edits contain instances in which a human editor has inserted a single phrase into, or deleted a contiguous phrase from, an existing sentence. |
| Outcome: | The data show that edits differ from the language observed in standard corpora and that models trained on edits encode different aspects of semantics and discourse than models trained in raw text. |
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022.emnlp-main)
Copied to clipboard
Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Minjoon Seo
| Challenge: | Language Models (LMs) become outdated as the world changes, a phenomenon called temporal misalignment. |
| Approach: | They propose a lifelong benchmark that utilizes the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation. |
| Outcome: | The proposed benchmark can be trained on the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation. |