Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for integrating knowledge graphs into pre-trained language models have been poorly implemented. |
| Approach: | They propose a self-supervised entity masking scheme that exploits relational knowledge underlying the text. |
| Outcome: | The proposed model achieves improved performance on five benchmarks, including question answering and knowledge base completion. |
Similar Papers
A Framework for Adapting Pre-Trained Language Models to Knowledge Graph Completion (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has demonstrated that entity representations can be extracted from pre-trained language models to develop knowledge graph completion models that are more robust to the naturally occurring sparsity found in knowledge graphs. |
| Approach: | They propose unsupervised and supervised methods to extract more informative representations from pre-trained language models to develop knowledge graph completion models. |
| Outcome: | The proposed model outperforms recent neural models in terms of performance and unsupervised processing methods. |
Self-supervised Graph Masking Pre-training for Graph-to-Text Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models (PLMs) have advanced Graph-to-Text generation by processing the linearised version of a graph. |
| Approach: | They propose to mask pre-training tasks that neither require supervision signals nor adjust the architecture of the underlying pre-trained encoder-decoder model. |
| Outcome: | The proposed method achieves state-of-the-art results on WebNLG+2020 and EventNarrative datasets and is very efficient in the low-resource setting. |
Graph Pre-training for AMR Parsing and Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Abstract meaning representation (AMR) highlights the core semantic information of text in a graph structure. |
| Approach: | They propose two graph auto-encoding strategies for graph-to-graph pre-training and four tasks to integrate text and graph information during pre-tuning to improve structure awareness. |
| Outcome: | The proposed model is superior to pre-trained language models on AMR parsing and AMR-to-text generation tasks. |
KLMo: Knowledge Graph Enhanced Pretrained Language Model with Fine-Grained Relationships (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge-enhanced pretrained language models focus on entity information and ignore fine-grained relationships between entities. |
| Approach: | They propose to incorporate KG into the language learning process to obtain a KG-enhanced pretrained Language Model. |
| Outcome: | The proposed model improves on several knowledge-driven tasks, such as entity typing and relation classification, compared with the state-of-the-art knowledge-enhanced PLMs. |
GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Recent improvements in KG-to-text generation are due to additional pre-training tasks . these tasks require extensive computational resources while only suggesting marginal improvements. |
| Approach: | They propose a mask structure to capture neighborhood information and a type encoder that adds a bias to the graph-attention weights depending on the connection type. |
| Outcome: | The proposed model outperforms state-of-the-art models while requiring no additional pre-training tasks. |
Accurate Text-Enhanced Knowledge Graph Representation Learning (N18-1)
Copied to clipboard
| Challenge: | Existing representation learning methods for knowledge graph representation do not consider the ambiguity of relations and entities. |
| Approach: | They propose a text-enhanced knowledge graph representation learning method which exploits the entity descriptions and triple-specific relation mention to enhance representations. |
| Outcome: | The proposed method outperforms existing representation learning models on link prediction and triple classification tasks and significantly outperformed existing models. |
Learning Knowledge-Enhanced Contextual Language Representations for Domain Natural Language Understanding (2023.emnlp-main)
Copied to clipboard
Taolin Zhang, Ruyao Xu, Chengyu Wang, Zhongjie Duan, Cen Chen, Minghui Qiu, Dawei Cheng, Xiaofeng He, Weining Qian
| Challenge: | Existing methods for pre-training KEPLMs with relational triples are difficult to adapt to close domains due to the lack of sufficient domain graph semantics. |
| Approach: | They propose a Knowledge-enhanced language representation learning framework for various closed domains that captures the implicit graph structure among the entities. |
| Outcome: | The proposed framework outperforms existing methods for pre-training KEPLMs in closed domains significantly. |
Enhancing Contextual Word Representations Using Embedding of Neighboring Entities in Knowledge Graphs (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for pre-trained language models lack explicit grounding in real-world entities. |
| Approach: | They propose a mechanism that integrates the structure of a KG into recent PLM architectures by generalizing the embeddings of neighboring entities. |
| Outcome: | The proposed method improves a classification task, entity typing task and language comprehension tasks. |
SKILL: Structured Knowledge Infusion for Large Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated human-level performance on a vast spectrum of natural language tasks. |
| Approach: | They propose a method to infuse structured knowledge into large language models by directly training T5 models on factual triples of knowledge graphs (KGs). |
| Outcome: | The proposed method outperforms baseline models on FreebaseQA and WikiHop, as well as the Wikidata-answerable subset of TriviaQA and NaturalQuestions. |
Contrastive Document Representation Learning with Graph Attention Networks (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for document representation learning are significantly affected by the scarcity of document-level data. |
| Approach: | They propose to use a graph attention network on top of the available pretrained Transformers model to learn document embeddings. |
| Outcome: | Empirically, the proposed approach is effective in document classification and document retrieval tasks. |