CLER: Cross-task Learning with Expert Representation to Generalize Reading and Understanding (D19-58)
Copied to clipboard
| Challenge: | In-domain datasets are used to train and validate our model, and other out-of-domain data are used for validation. |
| Approach: | They propose a model which uses cross-task learning with expert representation for the generalization of reading and understanding. |
| Outcome: | The proposed model achieved an average F1 score of 66.1 % in the out-of-domain setting, which is a 4.3 percentage point improvement over the official BERT baseline model. |
Similar Papers
Eliciting and Understanding Cross-task Skills with Task-level Mixture-of-Experts (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained transformer models are capable of multitasking on diverse NLP tasks, but little is known about how multitaskability and cross-task generalization is achieved. |
| Approach: | They propose to use a transformer-based mixture-of-expert model with a router component to choose among experts dynamically and flexibly. |
| Outcome: | The proposed models improve the average performance gain (ARG) metric by 2.6% when adapting to unseen tasks, and by 5.6% in zero-shot generalization settings. |
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (D19-58)
Copied to clipboard
| Challenge: | MRQA datasets have been used to benchmark progress in general-purpose language understanding. |
| Approach: | They propose to combine 18 question answering datasets into one shared task to evaluate their generalization capabilities. |
| Outcome: | The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than baseline based on BERT. |
Cross-Lingual Machine Reading Comprehension (D19-1)
Copied to clipboard
| Challenge: | Existing work on machine reading comprehension task is focused on English, but there are few efforts on other languages due to the lack of large-scale training data. |
| Approach: | They propose a cross-lingual machine reading comprehension task for other languages . they propose cloze-style reading comprehension and various neural network approaches . |
| Outcome: | The proposed model improves reading comprehension performance of Chinese datasets over state-of-the-art systems by a large margin over existing systems. |
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (P19-1)
Copied to clipboard
| Challenge: | A large number of reading comprehension (RC) datasets have been created, but little research has been done on whether they generalize to one another and the extent to which existing datasets can be leveraged for improving performance on new ones. |
| Approach: | They propose a BERT-based reading comprehension model that can be trained on multiple RC datasets. |
| Outcome: | The proposed model can be trained on multiple RC datasets and improve performance on five RC data. |
Knowledge-Empowered Representation Learning for Chinese Medical Reading Comprehension: Task, Model and Resources (2021.findings-acl)
Copied to clipboard
| Challenge: | MRC is a popular task in NLP, aiming to understand a passage and answer the relevant questions. |
| Approach: | They propose a multi-target machine learning task for the medical domain that predicts answers to medical questions and corresponding support sentences from medical information sources simultaneously. |
| Outcome: | The proposed model outperforms baselines by fusing context-aware and knowledge-awful token representations. |
Cross-lingual and Cross-domain Evaluation of Machine Reading Comprehension with Squad and CALOR-Quest Corpora (2020.lrec-1)
Copied to clipboard
| Challenge: | a recent study has shown that language mismatch and domain mismatch can affect performance of a machine reading task . a factor between language mismatched and domain-mismatched has the strongest influence on performance . |
| Approach: | They compare the cross-language and cross-domain capabilities of BERT on a machine reading comprehension task on two corpora: SQuAD and a new French Machine Reading dataset. |
| Outcome: | The proposed model matches human performance on a machine reading comprehension task with BERT on Chinese and French documents with interesting results. |
Cross-Task Knowledge Transfer for Query-Based Text Summarization (D19-58)
Copied to clipboard
| Challenge: | Existing methods for summarization data corpora are limited to extractive and abstractive summarizing. |
| Approach: | They propose to use machine reading comprehension (MRC) and query-based text summarization to produce extractive and abstractive summaries from pre-trained MRC and MT models. |
| Outcome: | The proposed model outperforms existing methods on CNN/Daily Mail and Debatepedia datasets and can be used as a baseline for future systems. |
Comprehensive Multi-Dataset Evaluation of Reading Comprehension (D19-58)
Copied to clipboard
| Challenge: | Recent research aims to facilitate training and evaluation on several reading comprehension datasets at the same time. |
| Approach: | They propose an evaluation server that reports performance on seven diverse reading comprehension datasets and includes synthetic augmentations to test models' ability to handle out-of-domain questions. |
| Outcome: | The evaluation server performs on seven reading comprehension datasets, and collects and includes synthetic augmentations for these datasets to test models' ability to handle out-of-domain questions. |
Generalizing Question Answering System with Pre-trained Language Model Fine-tuning (D19-58)
Copied to clipboard
| Challenge: | Existing methods focus on improving in-domain performance, leaving open the question of how they can generalize to out-of-domain and unseen RC tasks. |
| Approach: | They propose a multi-task learning framework that learns the shared representation across different tasks and builds on a large pre-trained language model and fine-tuned on multiple RC datasets. |
| Outcome: | The proposed framework improves the BERT-Large baseline by 8.39 and 7.22 respectively. |
Towards Interpreting BERT for Reading Comprehension Based QA (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models such as ELMO and XLNet have achieved state-of-the-art performance on various NLP tasks. |
| Approach: | They propose to define a layer’s role or functionality using Integrated Gradients and perform preliminary analysis across all layers. |
| Outcome: | The proposed model performs better than existing models on RCQA and ELMO, but it lacks the human-level performance needed to perform the task. |