The KITMUS Test: Evaluating Knowledge Integration from Multiple Sources (2023.acl-long)
Copied to clipboard
Akshatha Arodi, Martin Pömsl, Kaheer Suleman, Adam Trischler, Alexandra Olteanu, Jackie Chi Kit Cheung
| Challenge: | Existing models that make inferences using information from multiple sources are largely understudied . |
| Approach: | They propose a test suite of coreference resolution subtasks that require reasoning over multiple facts and introduce subtask where knowledge is present only at inference time using fictional knowledge. |
| Outcome: | The proposed subtasks differ in terms of which knowledge sources contain the relevant facts and where knowledge is present only at inference time using fictional knowledge. |
Similar Papers
What Has Been Enhanced in my Knowledge-Enhanced Language Model? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge integration methods such as linear probes and prompts have key limitations in answering these questions. |
| Approach: | They propose a new probe model which integrates external knowledge from knowledge graphs into pretrained language models (LMs) ERNIE and K-Adapter are proposed as KI methods . |
| Outcome: | The proposed model interprets two well-known KELMs using graph attention on the corresponding knowledge graph for interpretation. |
How Much Knowledge Can You Pack Into the Parameters of a Language Model? (2020.emnlp-main)
Copied to clipboard
| Challenge: | In this paper, we show that large neural language models trained on unstructured text can attain competitive results on open-domain question answering benchmarks without access to external knowledge. |
| Approach: | They propose to fine-tune pre-trained neural language models to answer questions without external knowledge . they show that this approach scales with model size and performs competitively . |
| Outcome: | The proposed approach scales with model size and performs competitively with open-domain systems that explicitly retrieve answers from an external knowledge source when answering questions. |
Pretrained Language Models for Dialogue Generation with Multiple Input Sources (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pretrained language models have achieved outstanding performance on natural language understanding tasks. |
| Approach: | They propose to fuse attention information from multiple input sources to achieve better relevance with dialogue history than simple fusion baselines. |
| Outcome: | The proposed models deliver higher relevance with dialogue history than baselines. |
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences (P18-2)
Copied to clipboard
| Challenge: | a new test set shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. |
| Approach: | They create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. |
| Outcome: | The new examples are simpler than the SNLI test set, but the state-of-the-art systems perform poorly on it. |
From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for natural language processing (NLP) are fine-tuned and released for research and deployments. |
| Approach: | They propose a PLM reuse paradigm that merges teacher-PLM knowledge into a student model. |
| Outcome: | The proposed paradigm can reduce the computational cost and environmental side-effects of retraining the PLM from scratch. |
LM-CORE: Language Models with Contextually Relevant External Knowledge (2022.findings-naacl)
Copied to clipboard
| Challenge: | Large pre-trained language models can capture factual knowledge in their parameters but storing large amounts of knowledge in the model parameters is sub-optimal given the ever-growing amounts of information and resource requirements. |
| Approach: | They propose a framework that provides explicit access to contextually relevant structured knowledge to the model and train it to use that knowledge. |
| Outcome: | The proposed framework outperforms state-of-the-art knowledge-enhanced language models on knowledge probing tasks and can handle knowledge updates. |
Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models (LMs) have shown effectiveness in literature understanding tasks, especially when tuned via contrastive learning. |
| Approach: | They propose a multi-task contrastive learning framework that enables common knowledge sharing across different scientific literature understanding tasks while preventing task-specific skills from interfering with each other. |
| Outcome: | The proposed framework outperforms state-of-the-art pre-trained language models on a comprehensive dataset. |
Probing Pretrained Language Models for Lexical Semantics (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on morphosyntactic, semantic, and world knowledge, but it remains unclear to what extent LMs derive lexical type-level knowledge from words in context. |
| Approach: | They propose to use multilingual and monolingual LMs to extract lexical type-level knowledge from words in context. |
| Outcome: | The proposed models perform well across six typologically diverse languages and five lexical tasks. |
Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models are trained on vast unlabeled data, rich in world knowledge. |
| Approach: | They propose a categorization scheme for factual probing methods based on how inputs, outputs and probed PLMs are adapted . they synthesize insights about knowledge retention and prompt optimization in PLM models and analyze obstacles to adopting them as knowledge bases . |
| Outcome: | The proposed method synthesizes insights about knowledge retention and prompt optimization in PLMs, analyzes obstacles to adopting them as knowledge bases and outline directions for future work. |
Estimating Knowledge in Large Language Models Without Generating a Single Token (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate knowledge in large language models require querying and evaluating the model's generated responses. |
| Approach: | They ask whether it is possible to estimate how knowledgeable a model is about a subject entity only from its internal computation. |
| Outcome: | The proposed model performs well with QA accuracy and FActScore . it can be leveraged to guide decisions on how to apply further training or augment queries with retrieval. |