Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension (P19-1)
Copied to clipboard
| Challenge: | Recent results show pre-trained language models (LMs) can improve machine reading comprehension (MRC) Experimental results indicate that KT-NET offers significant and consistent improvements over BERT . |
| Approach: | They propose a method that leverages external knowledge bases to improve machine reading comprehension (MRC) KT-NET employs an attention mechanism to select desired knowledge from KBs and fuses selected knowledge with BERT to enable context- and knowledge-aware predictions. |
| Outcome: | The proposed model outperforms baseline models on ReCoRD and SQuAD1.1 benchmarks and ranks 1st on the ReCoDR and SQUAD1.1 leaderboards. |
Similar Papers
REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models have achieved great success on Machine Reading Comprehension (MRC) however, the poor support in evidence extraction hinders them from further advancing MRC. |
| Approach: | They propose a REtrieval-based pre-training approach that strengthens evidence extraction during pre-training by inherited downstream MRC tasks. |
| Outcome: | The proposed approach strengthens evidence extraction during pre-training, which is further inherited by downstream tasks. |
IIT-KGP at COIN 2019: Using pre-trained Language Models for modeling Machine Comprehension (D19-60)
Copied to clipboard
| Challenge: | Using pre-trained language models, we can model machine comprehension using commonsense reasoning. |
| Approach: | They propose a machine comprehension model that leverages pre-trained language models over commonsense knowledge bases. |
| Outcome: | The proposed model improves on baseline models and other commonsense knowledge bases. |
Improving Machine Reading Comprehension with General Reading Strategies (N19-1)
Copied to clipboard
| Challenge: | Recent studies have shown that reading strategies improve comprehension levels for readers lacking adequate prior knowledge. |
| Approach: | They propose three general strategies to improve machine reading comprehension (MRC) by fine-tuning a pre-trained model with strategies and a target task. |
| Outcome: | The proposed models improve non-extractive machine reading comprehension (MRC) on the largest general domain multiple-choice dataset RACE. |
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (N19-1)
Copied to clipboard
| Challenge: | Existing language representation models pre-train deep bidirectional representations from unlabeled text without significant task-specific architecture modifications. |
| Approach: | They propose a language representation model that pre-trains bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. |
| Outcome: | The proposed model achieves state-of-the-art results on eleven natural language processing tasks, pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement) |
Cross-Lingual Machine Reading Comprehension (D19-1)
Copied to clipboard
| Challenge: | Existing work on machine reading comprehension task is focused on English, but there are few efforts on other languages due to the lack of large-scale training data. |
| Approach: | They propose a cross-lingual machine reading comprehension task for other languages . they propose cloze-style reading comprehension and various neural network approaches . |
| Outcome: | The proposed model improves reading comprehension performance of Chinese datasets over state-of-the-art systems by a large margin over existing systems. |
Span Selection Pre-training for Question Answering (2020.acl-main)
Copied to clipboard
Michael Glass, Alfio Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, G P Shrivatsa Bhargav, Dinesh Garg, Avi Sil
| Challenge: | Pre-trained BERTs provide large gains across many language understanding tasks, achieving a new state-of-the-art (SOTA). |
| Approach: | They propose a new pre-training task inspired by reading comprehension to better align the pre- training from memorization to understanding. |
| Outcome: | The proposed model outperforms BERT-BASE and BERT LARGE on a new dataset and improves answer prediction F1 by 4 points and supporting fact prediction F1. |
Knowledge-Empowered Representation Learning for Chinese Medical Reading Comprehension: Task, Model and Resources (2021.findings-acl)
Copied to clipboard
| Challenge: | MRC is a popular task in NLP, aiming to understand a passage and answer the relevant questions. |
| Approach: | They propose a multi-target machine learning task for the medical domain that predicts answers to medical questions and corresponding support sentences from medical information sources simultaneously. |
| Outcome: | The proposed model outperforms baselines by fusing context-aware and knowledge-awful token representations. |
D-NET: A Pre-Training and Fine-Tuning Framework for Improving the Generalization of Machine Reading Comprehension (D19-58)
Copied to clipboard
Hongyu Li, Xiyuan Zhang, Yibing Liu, Yiming Zhang, Quan Wang, Xiangyang Zhou, Jing Liu, Hua Wu, Haifeng Wang
| Challenge: | MRC requires machines to understand text and answer questions about the text. |
| Approach: | They propose a simple system Baidu submitted for MRQA 2019 Shared Task that focused on generalization of machine reading comprehension (MRC) models. |
| Outcome: | The proposed system is ranked at top 1 of all participants in terms of averaged F1 score. |
ERNIE: Enhanced Language Representation with Informative Entities (P19-1)
Copied to clipboard
| Challenge: | Existing pre-trained language models rarely consider incorporating knowledge graphs (KGs) Existing models capture rich semantic patterns from plain text and can be fine-tuned to improve performance of NLP tasks. |
| Approach: | They propose to incorporate knowledge graphs into pre-trained language models to enhance language representation with external knowledge. |
| Outcome: | The proposed model can take full advantage of lexical, syntactic, and knowledge information simultaneously. |
Machine Reading Comprehension Using Structural Knowledge Graph-aware Network (D19-1)
Copied to clipboard
| Challenge: | Recent large-scale datasets specify that external knowledge is required to answer questions. |
| Approach: | They propose a model that leverages external knowledge to construct sub-graphs for entities in machine comprehension context. |
| Outcome: | The proposed model achieves state-of-the-art performance on the ReCoRD dataset. |