LM-CORE: Language Models with Contextually Relevant External Knowledge (2022.findings-naacl)
Copied to clipboard
| Challenge: | Large pre-trained language models can capture factual knowledge in their parameters but storing large amounts of knowledge in the model parameters is sub-optimal given the ever-growing amounts of information and resource requirements. |
| Approach: | They propose a framework that provides explicit access to contextually relevant structured knowledge to the model and train it to use that knowledge. |
| Outcome: | The proposed framework outperforms state-of-the-art knowledge-enhanced language models on knowledge probing tasks and can handle knowledge updates. |
Similar Papers
How Much Knowledge Can You Pack Into the Parameters of a Language Model? (2020.emnlp-main)
Copied to clipboard
| Challenge: | In this paper, we show that large neural language models trained on unstructured text can attain competitive results on open-domain question answering benchmarks without access to external knowledge. |
| Approach: | They propose to fine-tune pre-trained neural language models to answer questions without external knowledge . they show that this approach scales with model size and performs competitively . |
| Outcome: | The proposed approach scales with model size and performs competitively with open-domain systems that explicitly retrieve answers from an external knowledge source when answering questions. |
Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries (2021.eacl-main)
Copied to clipboard
| Challenge: | Pretrained language models have been suggested as an alternative or complement to structured knowledge bases . however, this paradigm has only been considered in a very limited setting . |
| Approach: | They propose a paradigm that allows LMs to store a large number of entities . they propose LM-as-KB paradigm which allows querying stored facts . |
| Outcome: | The proposed paradigm allows handling 21k entities whose name is found in common LM vocabularies . the proposed paradigm has only been considered in a very limited setting . |
Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for updating knowledge show little propagation of injected knowledge. |
| Approach: | They propose to inject individual facts into LMs and evaluate whether they can propagate injected facts while not changing predictions on other contexts. |
| Outcome: | The proposed model can make inferences based on injected facts and propagate them . existing methods show little propagation of injected knowledge . |
Language Models as Knowledge Bases? (D19-1)
Copied to clipboard
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller
| Challenge: | Recent advances in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. |
| Approach: | They present a method for pretraining language models on large textual corpora . they find that they can store relational knowledge and answer queries structured as "fill-in-the-blank" queries. |
| Outcome: | The proposed language models can recall factual knowledge without fine-tuning without fine tuning . the proposed models can answer queries structured as "fill-in-the-blank" cloze statements . |
On the Importance of Effectively Adapting Pretrained Language Models for Active Learning (2022.acl-short)
Copied to clipboard
| Challenge: | Recent active learning approaches in NLP use off-the-shelf pretrained language models (LMs) . a poor training strategy can be catastrophic for AL, authors argue . |
| Approach: | They propose to first adapt the pretrained LM to the target task and then use it for AL. |
| Outcome: | The proposed approach provides substantial data efficiency improvements compared to the standard fine-tuning approach. |
How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have focused on enhancing the factualness of large language models using context knowledge. |
| Approach: | They propose to use ChatGPT to construct probing datasets that provide diverse and coherent evidence corresponding to various facts. |
| Outcome: | The proposed model can encode knowledge across different layers, and it is compared with existing models. |
Do Language Embeddings capture Scales? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained Language Models possess significant linguistic, common sense and factual knowledge, but are short of the capability required for general common-sense reasoning. |
| Approach: | They propose to train pretrained language models with a method of canonicalizing numbers . they address a task which is also pre-requisite for general common-sense reasoning . |
| Outcome: | The proposed model can answer questions about common sense and linguistics, but lacks the capability to answer questions on scalar attributes. |
Probing Pretrained Language Models for Lexical Semantics (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on morphosyntactic, semantic, and world knowledge, but it remains unclear to what extent LMs derive lexical type-level knowledge from words in context. |
| Approach: | They propose to use multilingual and monolingual LMs to extract lexical type-level knowledge from words in context. |
| Outcome: | The proposed models perform well across six typologically diverse languages and five lexical tasks. |
Neural Natural Language Inference Models Enhanced with External Knowledge (P18-1)
Copied to clipboard
| Challenge: | Existing datasets that allow for complex models to be trained are limited . if data is not available, can machines learn all knowledge needed to perform natural language inference? |
| Approach: | They propose to enrich neural natural language inference models with external knowledge . they propose to use this knowledge to build NLI models to leverage it . |
| Outcome: | The proposed models improve on the SNLI and MultiNLI datasets. |
LMSOC: An Approach for Socially Sensitive Pretraining (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pretraining models have been shown to learn effective linguistic representations for many NLP tasks, but there are many real-world contextual aspects of language that current approaches do not capture. |
| Approach: | They propose to integrate speaker social context into the learned representations of large-scale language models by using graph representation learning algorithms and primed language model pretraining with these social context representations. |
| Outcome: | The proposed approach improves on geographically sensitive language modeling tasks by more than 100% relative lift on MRR compared to baselines. |