Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for language model pretraining use limited knowledge graph data for knowledge-intensive tasks. |
| Approach: | They propose to make better use of multilingual annotations and language agnostic properties of KG triples for pretraining LMs. |
| Outcome: | The proposed models show significant performance improvements on a wide range of knowledge-intensive cross-lingual tasks. |
Similar Papers
Adapters for Enhanced Modeling of Multilingual Knowledge and Text (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models learn facts from text corpora, but knowledge graphs contain facts in an explicit triple format, restricting their research and application. |
| Approach: | They propose to enhance multilingual language models with knowledge from multilingual knowledge graphs . they propose to use cross-lingual entity alignment and facts from MLKGs to improve performance . |
| Outcome: | The proposed model improves MLLMs with cross-lingual entity alignment and facts from multilingual knowledge graphs for many languages while maintaining performance on other general language tasks. |
Prix-LM: Pretraining for Multilingual Knowledge Base Construction (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to build and enrich multilingual knowledge bases have not been successful . knowledge expressed in different languages may be complementary and unequally distributed . |
| Approach: | They propose a model that integrates useful multilingual and KB-based factual knowledge into a single model. |
| Outcome: | The proposed model can provide richer combined knowledge than monolingual KBs. |
SKILL: Structured Knowledge Infusion for Large Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated human-level performance on a vast spectrum of natural language tasks. |
| Approach: | They propose a method to infuse structured knowledge into large language models by directly training T5 models on factual triples of knowledge graphs (KGs). |
| Outcome: | The proposed method outperforms baseline models on FreebaseQA and WikiHop, as well as the Wikidata-answerable subset of TriviaQA and NaturalQuestions. |
Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing work on data-to-text generation focused on domain-specific benchmark datasets. |
| Approach: | They use a KG-Wikipedia text aligned corpus to verbalize the entire English Wikidata KG . they show that this approach can be used to integrate structured KGs and natural language corpora . |
| Outcome: | The proposed method improves on open domain QA and the LAMA knowledge probe. |
Learning Knowledge-Enhanced Contextual Language Representations for Domain Natural Language Understanding (2023.emnlp-main)
Copied to clipboard
Taolin Zhang, Ruyao Xu, Chengyu Wang, Zhongjie Duan, Cen Chen, Minghui Qiu, Dawei Cheng, Xiaofeng He, Weining Qian
| Challenge: | Existing methods for pre-training KEPLMs with relational triples are difficult to adapt to close domains due to the lack of sufficient domain graph semantics. |
| Approach: | They propose a Knowledge-enhanced language representation learning framework for various closed domains that captures the implicit graph structure among the entities. |
| Outcome: | The proposed framework outperforms existing methods for pre-training KEPLMs in closed domains significantly. |
Retrieval and Reasoning on KGs: Integrate Knowledge Graphs into Large Language Models for Complex Question Answering (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have performed impressively in various NLP tasks, but their inherent hallucination phenomena severely challenge their credibility in complex reasoning. |
| Approach: | They propose to integrate explainable Knowledge Graphs (KGs) with LLMs to alleviate hallucinations . they construct subgraphs to enhance the retrieval capabilities of KGs via CoT reasoning. |
| Outcome: | Extensive experiments on two KGQA datasets show that the proposed model achieves convincing performance compared to strong baselines. |
Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in language modeling have led to the emergence of large language models capable ofvarious natural language processing tasks. |
| Approach: | They propose a multi-instructional training approach that integrates a large language model with a speech encoder to harness the capabilities of LLMs for speech recognition and beyond. |
| Outcome: | The proposed model can be trained and aligned with a multilingual LLM on 1900 hours of transcribed data from 139 languages. |
KLMo: Knowledge Graph Enhanced Pretrained Language Model with Fine-Grained Relationships (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge-enhanced pretrained language models focus on entity information and ignore fine-grained relationships between entities. |
| Approach: | They propose to incorporate KG into the language learning process to obtain a KG-enhanced pretrained Language Model. |
| Outcome: | The proposed model improves on several knowledge-driven tasks, such as entity typing and relation classification, compared with the state-of-the-art knowledge-enhanced PLMs. |
Learning beyond Datasets: Knowledge Graph Augmented Neural Networks for Natural Language Processing (N18-1)
Copied to clipboard
| Challenge: | Currently, machine learning is limited in scalability and is limited to specific training data. |
| Approach: | They propose to enhance learning models with world knowledge in the form of Knowledge Graph fact triples for natural language processing tasks. |
| Outcome: | The proposed method is highly scalable to the amount of prior information that has to be processed and can be applied to any generic NLP task. |
Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean (2024.lrec-main)
Copied to clipboard
ChangSu Choi, Yongbin Jeong, Seoyoon Park, Inho Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansaem Kim, KyungTae Lim
| Challenge: | Large language models (LLMs) use pretraining to predict the subsequent word, but less-resourced languages are being overlooked. |
| Approach: | They propose to expand the MLLM vocabularies to enhance expressiveness and use bilingual data for pretraining to align the high- and less-resourced languages. |
| Outcome: | The proposed model outperforms existing models in qualitative analyses compared to Korean monolingual models. |