Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller
| Challenge: | Recent advances in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. |
| Approach: | They present a method for pretraining language models on large textual corpora . they find that they can store relational knowledge and answer queries structured as "fill-in-the-blank" queries. |
| Outcome: | The proposed language models can recall factual knowledge without fine-tuning without fine tuning . the proposed models can answer queries structured as "fill-in-the-blank" cloze statements . |
Similar Papers
Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA? (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work is limited in using small benchmarks with high test-train overlaps. |
| Approach: | They construct a dataset of closed-book QA using SQuAD and investigate the performance of BART. |
| Outcome: | Experiments show that pre-trained language models can achieve high performance on closed-book QA tasks. |
How Much Knowledge Can You Pack Into the Parameters of a Language Model? (2020.emnlp-main)
Copied to clipboard
| Challenge: | In this paper, we show that large neural language models trained on unstructured text can attain competitive results on open-domain question answering benchmarks without access to external knowledge. |
| Approach: | They propose to fine-tune pre-trained neural language models to answer questions without external knowledge . they show that this approach scales with model size and performs competitively . |
| Outcome: | The proposed approach scales with model size and performs competitively with open-domain systems that explicitly retrieve answers from an external knowledge source when answering questions. |
Are Visual-Linguistic Models Commonsense Knowledge Bases? (2022.coling-1)
Copied to clipboard
| Challenge: | PTLMs are used to extract knowledge from text on demand. |
| Approach: | They compare visual-linguistic and language-only visual-language models in a zero-shot commonsense question answering inference task. |
| Outcome: | The proposed models are highly promising on certain types of commonsense knowledge associated with the visual world. |
On the Nature of BERT: Correlating Fine-Tuning and Linguistic Competence (2022.coling-1)
Copied to clipboard
| Challenge: | Several studies on the interpretation of Neural Language Models (NLMs) focus on the linguistic generalization abilities of pre-trained models, but little attention is paid to how the linguistic knowledge of the models changes during fine-tuning. |
| Approach: | They propose to examine whether a wide range of linguistic phenomena are forgotten during fine-tuning and whether it is possible to predict the fine- tuned accuracy solely relying on the assessed linguistic competence. |
| Outcome: | The proposed model can predict the evolution of written language competence of native language learners based on the assessed linguistic competence. |
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |
Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | Pre-trained language models are language models that are pre-taught on large-scaled corpora in a self-supervised fashion. |
| Approach: | This tutorial provides a broad and comprehensive introduction to pre-trained language models . it focuses on emerging methods that enable PLMs to perform diverse downstream tasks . |
| Outcome: | This tutorial focuses on the benefits of pre-trained language models and how to use them in NLP tasks. |
Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries (2021.eacl-main)
Copied to clipboard
| Challenge: | Pretrained language models have been suggested as an alternative or complement to structured knowledge bases . however, this paradigm has only been considered in a very limited setting . |
| Approach: | They propose a paradigm that allows LMs to store a large number of entities . they propose LM-as-KB paradigm which allows querying stored facts . |
| Outcome: | The proposed paradigm allows handling 21k entities whose name is found in common LM vocabularies . the proposed paradigm has only been considered in a very limited setting . |
Probing Pretrained Language Models for Lexical Semantics (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on morphosyntactic, semantic, and world knowledge, but it remains unclear to what extent LMs derive lexical type-level knowledge from words in context. |
| Approach: | They propose to use multilingual and monolingual LMs to extract lexical type-level knowledge from words in context. |
| Outcome: | The proposed models perform well across six typologically diverse languages and five lexical tasks. |
Can LMs Store and Retrieve 1-to-N Relational Knowledge? (2023.acl-srw)
Copied to clipboard
| Challenge: | Pretraining language models on large amounts of text has made it difficult to store and retrieve world knowledge. |
| Approach: | They propose to view pretrained language models as knowledge bases by examining their ability to store and retrieve world knowledge. |
| Outcome: | The proposed model can store and retrieve world knowledge with high accuracy, but it is not clear how accurately it can handle 1-to-N relational knowledge. |
Static Embeddings as Efficient Knowledge Bases? (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent research investigates factual knowledge stored in large pretrained language models . masked sentences such as “Paris is the capital of [MASK]” are used as probes . |
| Approach: | They use masked sentences to test whether a language model can capture factual knowledge . they show that static embeddings perform better than PLMs when restricted to a candidate set . |
| Outcome: | The results show that static embeddings perform better than PLMs when restricted to a candidate set . |