Parameter-Efficient Domain Knowledge Integration from Multiple Sources for Biomedical Pre-trained Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing domain-specific pre-trained language models (PLMs) rely on self-supervised learning over large amounts of domain text, without explicitly integrating domain- specific knowledge. |
| Approach: | They propose to integrate domain knowledge from diverse sources into PLMs by using adapters that are pre-trained for individual domain knowledge sources and integrated via an attention-based knowledge controller. |
| Outcome: | The proposed architecture integrates domain knowledge from diverse sources into PLMs in a parameter-efficient way. |
Similar Papers
Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models’ Memories (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models demonstrate excellent abilities to understand texts in the generic domain while struggling in a specific domain. |
| Approach: | They propose to decouple the feed-forward networks of the Transformer architecture into two parts to maintain old-domain knowledge and a mixture-of-adapters gate to inject domain-specific knowledge in parallel. |
| Outcome: | The proposed method achieves superior performance on in-domain, out-of-domain and knowledge-intensive tasks. |
Can Language Models be Biomedical Knowledge Bases? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on probing LMs in the general domain but little attention has been given to whether they can be used as domain knowledge bases. |
| Approach: | They propose to use 49K biomedical factual knowledge triples to probe LMs for biomedically . they find that biomedic LM can achieve up to 18.51% Acc@5 on retrieving biomedcial knowledge. |
| Outcome: | The proposed biomedical factual knowledge probing benchmark achieves 18.51% Acc@5 on biomedically-relevant knowledge retrieval. |
Efficient Hierarchical Domain Adaptation for Pretrained Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to allow domain adaptation to diverse domains are expensive and require continuing training in-domain. |
| Approach: | They propose a method to permit domain adaptation to many diverse domains using a computationally efficient adapter approach. |
| Outcome: | The proposed method allows domain adaptation to many diverse domains while avoiding negative interference between unrelated domains. |
Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Survey (2025.findings-emnlp)
Copied to clipboard
| Challenge: | specialized LLMs are often limited in domain-specific applications that require specialized knowledge. |
| Approach: | They provide a comprehensive overview of four key methods to enhance large language models by integrating domain-specific knowledge. |
| Outcome: | The proposed methods are categorized into four key approaches: dynamic knowledge injection, static knowledge embedding, modular adapters, and prompt optimization. |
Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new method for continual pretraining transformer encoder models is proposed for specialized domains with limited training data. |
| Approach: | They propose to use LLM-generated data to enrich domain-specific ontologies and pretrain transformer encoder models as an ontology-informed embedding model for concept definitions. |
| Outcome: | The proposed method improves on standard MLM pretraining on invasion biology domains. |
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (2021.findings-acl)
Copied to clipboard
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, Ming Zhou
| Challenge: | Existing methods for injecting knowledge into pre-trained models are inconsistent and can flush out knowledge when multiple kinds of knowledge are injected. |
| Approach: | They propose a framework that retains the original parameters of pre-trained models fixed and supports the development of versatile knowledge-infused models. |
| Outcome: | The proposed framework retains the original parameters of the pre-trained model fixed and supports the development of versatile knowledge-infused models. |
UDAPTER - Efficient Domain Adaptation Using Adapters (2023.eacl-main)
Copied to clipboard
| Challenge: | Using adapters, unsupervised domain adaptation (UDA) is more parameter efficient and requires large-scale data to be effective. |
| Approach: | They propose to add small bottleneck layers to each layer of a pre-trained language model to make it more parameter efficient by adding adapters. |
| Outcome: | The proposed methods outperform unsupervised domain adaptation methods such as DANN and DSN in natural language inference and sentiment classification tasks. |
Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linking (2021.acl-short)
Copied to clipboard
| Challenge: | Existing work on transferring domain-specific knowledge from a pretraining model to a resource-poor language is limited to English . a novel cross-lingual biomedical entity linking task is proposed to improve this capability. |
| Approach: | They propose a cross-lingual biomedical entity linking task and establish a new benchmark spanning 10 typologically diverse languages. |
| Outcome: | The proposed methods yield consistent gains across all target languages, sometimes up to 20 Precision@1 points, without any in-domain knowledge in the target language and without any parallel data. |
Generative Biomedical Entity Linking via Knowledge Base-Guided Pre-training and Synonyms-Aware Fine-tuning (2022.naacl-main)
Copied to clipboard
| Challenge: | Generative methods for biomedical entity linking (EL) use synonyms knowledge from knowledge bases (KB) this is not trivial to inject into a generative method, but it is cost-effective. |
| Approach: | They propose to inject synonyms knowledge into a generative model of biomedical EL by constructing synthetic samples with synonyms and definitions from KB and requiring the model to recover concept names. |
| Outcome: | The proposed method achieves state-of-the-art results on several biomedical EL tasks without candidate selection. |
Know-Adapter: Towards Knowledge-Aware Parameter-Efficient Transfer Learning for Few-shot Named Entity Recognition (2024.lrec-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a fundamental task in natural language processing. |
| Approach: | They propose a knowledgeable adapter to incorporate structure and semantic knowledge of knowledge graphs into PLMs for few-shot NER. |
| Outcome: | The proposed adapter improves the quality of retrieved information by adding explicit knowledge from external sources to PEFTs. |