How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis (2022.findings-acl)
Copied to clipboard
Shaobo Li, Xiaoguang Li, Lifeng Shang, Zhenhua Dong, Chengjie Sun, Bingquan Liu, Zhenzhou Ji, Xin Jiang, Qun Liu
| Challenge: | Recent studies show that pre-trained language models can fill in the missing factual words in cloze-style prompts such as ”Dante was born in [MASK]” . |
| Approach: | They propose to quantitatively measure and evaluate the word-level patterns that PLMs depend on to generate the missing factual words. |
| Outcome: | The proposed model fills in the missing factual words in cloze-style prompts by relying on effective clues or shortcut patterns. |
Similar Papers
Pre-training Language Models with Deterministic Factual Knowledge (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that Pre-trained Language Models fail to capture factual knowledge robustly. |
| Approach: | They propose to let PLMs learn the deterministic relationship between context and masked content to improve their ability to capture factual knowledge. |
| Outcome: | The proposed methods improve accuracy and consistency of factual knowledge capturing and boost performance of other knowledge-intensive tasks. |
Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models are trained on vast unlabeled data, rich in world knowledge. |
| Approach: | They propose a categorization scheme for factual probing methods based on how inputs, outputs and probed PLMs are adapted . they synthesize insights about knowledge retention and prompt optimization in PLM models and analyze obstacles to adopting them as knowledge bases . |
| Outcome: | The proposed method synthesizes insights about knowledge retention and prompt optimization in PLMs, analyzes obstacles to adopting them as knowledge bases and outline directions for future work. |
Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained masked language models can be factual knowledge bases. |
| Approach: | They conduct a rigorous study to explore the underlying predicting mechanisms of MLMs . they find that previous decent performance mainly owes to the biased prompts which overfit dataset artifacts a . |
| Outcome: | The proposed model improves on illustrative cases and external contexts . the results question the previous findings that MLMs can be reliable factual knowledge bases . |
Does the Correctness of Factual Knowledge Matter for Factual Knowledge-Enhanced Pre-trained Language Models? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work neither proves that pre-trained models successfully learn the injected factual knowledge nor proves there is a causal relation between injected knowledge and downstream performance improvements. |
| Approach: | They propose a counterfactual-based analysis framework to explore the causal effects of factual knowledge injection on the performance of language models within pretrain-finetune paradigm. |
| Outcome: | The proposed framework shows that factual knowledge injection is successful but correctness of injected knowledge only has limited effect on the models’ downstream performance. |
Measuring and Improving Consistency in Pretrained Language Models (2021.tacl-1)
Copied to clipboard
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, Yoav Goldberg
| Challenge: | In this paper, we examine whether pretrained language models are consistent with factual knowledge. |
| Approach: | They propose a method to improve consistency of pretrained language models . consistency is a desirable property of a good language understanding model, they argue . |
| Outcome: | The proposed model improves consistency and shows that it is effective. |
The Queen of England is not England’s Queen: On the Lack of Factual Coherency in PLMs (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing work evaluated how often PLMs can correctly predict a subject and a relation . previous work focused on evaluating how much PLM know, but this study focused on the internal state of knowledge inside them. |
| Approach: | They examine how often PLMs can correctly predict a subject and a relation . they also examine how knowledge inside PLM is embodied in the internal state . |
| Outcome: | The proposed model improves on the accuracy of the evidence paragraphs and manually written prompts. |
Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | Pre-trained language models are language models that are pre-taught on large-scaled corpora in a self-supervised fashion. |
| Approach: | This tutorial provides a broad and comprehensive introduction to pre-trained language models . it focuses on emerging methods that enable PLMs to perform diverse downstream tasks . |
| Outcome: | This tutorial focuses on the benefits of pre-trained language models and how to use them in NLP tasks. |
Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for updating knowledge show little propagation of injected knowledge. |
| Approach: | They propose to inject individual facts into LMs and evaluate whether they can propagate injected facts while not changing predictions on other contexts. |
| Outcome: | The proposed model can make inferences based on injected facts and propagate them . existing methods show little propagation of injected knowledge . |
On the Reliability of Large Language Models for Causal Discovery (2025.acl-long)
Copied to clipboard
| Challenge: | Existing statistical methods to identify causal relationships from observational data remain elusive. |
| Approach: | They examine the impact of memorization for accurate causal relation prediction, the influence of incorrect causal relations in pre-training data and the contextual nuances that influence LLMs’ understanding of causal relations. |
| Outcome: | The proposed models are effective in recognizing causal relations that occur frequently in pre-training data, but their ability to generalize to new or rare causal relations is limited. |
Calibrating Factual Knowledge in Pretrained Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that Pretrained Language Models can store factual knowledge, but facts stored in PLMs are not always correct. |
| Approach: | They propose a lightweight method to calibrate factual knowledge in PLMs without re-training from scratch. |
| Outcome: | The proposed method can be used to calibrate factual knowledge in PLMs without re-training from scratch. |