Fact Recall, Heuristics or Pure Guesswork? Precise Interpretations of Language Models for Fact Completion

Challenge:	Language models (LMs) can make a correct prediction based on many possible signals in a prompt, but not all corresponding to recall of factual associations.
Approach:	They propose a model-specific recipe for constructing datasets with examples of four different prediction scenarios: generic language modeling, guesswork, heuristics recall and exact fact recall.
Outcome:	The proposed model-specific recipe yields distinct results for each scenario.

Similar Papers

Overview

Challenge

Approach

Outcome

How Can We Know What Language Models Know? (2020.tacl-1)

Copied to clipboard

Zhengbao Jiang, Frank F. Xu, Jun Araki, Graham Neubig

Language Models

Challenge:	Recent work examines knowledge contained in language models by having the LM fill in the blanks of prompts such as “Obama is a __ by profession”.
Approach:	They propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts.
Outcome:	The proposed methods improve accuracy from 31.1% to 39.6% on the LAMA benchmark for extracting relational knowledge from LMs.

Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models (2023.findings-emnlp)

Copied to clipboard

Paul Youssef, Osman Koraş, Meijie Li, Jörg Schlötterer, Christin Seifert

Semantic Text Processing

Challenge:	Pre-trained language models are trained on vast unlabeled data, rich in world knowledge.
Approach:	They propose a categorization scheme for factual probing methods based on how inputs, outputs and probed PLMs are adapted . they synthesize insights about knowledge retention and prompt optimization in PLM models and analyze obstacles to adopting them as knowledge bases .
Outcome:	The proposed method synthesizes insights about knowledge retention and prompt optimization in PLMs, analyzes obstacles to adopting them as knowledge bases and outline directions for future work.

Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge (2024.eacl-long)

Copied to clipboard

Xin Zhao, Naoki Yoshinaga, Daisuke Oba

Semantic Text Processing

Challenge:	Using low-resource languages, multilingual language models (ML-LMs) have been developed to transfer factual knowledge across languages.
Approach:	They ask how ML-LMs acquire and represent factual knowledge . they use a multilingual factual information probing dataset to investigate ML .
Outcome:	The findings highlight the challenge of maintaining consistency factual knowledge across languages.

Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline (2025.emnlp-main)

Copied to clipboard

Meng Lu, Ruochen Zhang, Carsten Eickhoff, Ellie Pavlick

Multilinguality

Challenge:	Multilingual large language models (LLMs) exhibit factual inconsistencies across languages . authors identify two primary sources of error: insufficient engagement of reliable English-centric mechanism for factual recall, and incorrect translation from English back into the target language for the final answer.
Approach:	They propose two vector interventions to redirect the model toward better internal paths for higher factual consistency.
Outcome:	The proposed interventions increase the recall accuracy by over 35 percent for the lowest-performing language.

X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020.emnlp-main)

Copied to clipboard

Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, Graham Neubig

Semantic Text Processing

Challenge:	Language models (LMs) capture factual knowledge by filling in the blanks of cloze-style prompts.
Approach:	They propose a code-switching-based method to improve the ability of multilingual LMs to access knowledge and verify its effectiveness on several benchmark languages.
Outcome:	The proposed method improves the ability of multilingual LMs to access knowledge and verify its effectiveness on several benchmark languages.

Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases (2021.acl-long)

Copied to clipboard

Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, Jin Xu

Semantic Text Processing

Challenge:	Recent studies show that pre-trained masked language models can be factual knowledge bases.
Approach:	They conduct a rigorous study to explore the underlying predicting mechanisms of MLMs . they find that previous decent performance mainly owes to the biased prompts which overfit dataset artifacts a .
Outcome:	The proposed model improves on illustrative cases and external contexts . the results question the previous findings that MLMs can be reliable factual knowledge bases .

Dissecting Recall of Factual Associations in Auto-Regressive Language Models (2023.emnlp-main)

Copied to clipboard

Mor Geva, Jasmijn Bastings, Katja Filippova, Amir Globerson

Semantic Text Processing

Challenge:	Existing studies have focused on identifying where factual knowledge is encoded in the network, but little is known about how it is extracted from the model parameters during inference.
Approach:	They examine how factual associations are stored and retrieved internally in LMs . they use attention edges to identify critical points where information propagates to the prediction .
Outcome:	The proposed model aggregates information about subject and relation to predict the correct attribute . the model “queries” the enriched subject to extract the attribute based on the proposed model .

Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models (2025.acl-long)

Copied to clipboard

Mingyang Wang, Heike Adel, Lukas Lange, Yihong Liu, Ercong Nie, Jannik Strötgen, Hinrich Schuetze

Language Models

Challenge:	Multilingual language models store factual knowledge across languages but struggle to provide consistent responses to semantically equivalent prompts in different languages.
Approach:	They propose a linear shortcut method that bypasses computations in the final layers . this method improves accuracy and cross-lingual consistency .
Outcome:	The proposed method improves prediction accuracy and cross-lingual consistency.

Knowing the Facts but Choosing the Shortcut: Understanding How Large Language Models Compare Entities (2026.eacl-long)

Copied to clipboard

Hans Hergen Lehmann, Jae Hee Lee, Steven Schockaert, Stefan Wermter

Semantic Text Processing

Challenge:	Large Language Models (LLMs) are increasingly used for knowledge-based reasoning tasks, yet understanding when they rely on genuine knowledge versus superficial heuristics remains challenging.
Approach:	They propose to ask LLMs to compare numerical attributes to find out which country has the highest population, France or Germany.
Outcome:	The proposed model comparisons show that heuristics override principled reasoning for larger models, while smaller models show no discrimination.

How Do Multilingual Language Models Remember Facts? (2025.findings-acl)

Copied to clipboard

Constanza Fierro, Negar Foroutan, Desmond Elliott, Anders Søgaard

Semantic Text Processing

Challenge:	Prior research has focused on English monolingual models, but how these mechanisms generalize to non-English languages remains unexplored.
Approach:	They analyze three multilingual LLMs to find out how they can generalize recall mechanisms . they find that subject enrichment is language-independent, object extraction is language dependent .
Outcome:	The proposed model performs better in multilingual contexts than in English models . the model is more efficient in multi-lingual context, but it is more complex in multilinguistic models compared to English models.