Active Reading Comprehension: A Dataset for Learning the Question-Answer Relationship Strategy (P19-2)
Copied to clipboard
| Challenge: | Literature in quality learning suggests that task performance should also be evaluated on the undergone process to answer. |
| Approach: | They propose to use the Question-Answer Relationship (QAR) to evaluate a reader's ability to select different sources of information depending on the question type. |
| Outcome: | The proposed model will be used to evaluate reading comprehension with weak supervision. |
Similar Papers
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (P19-1)
Copied to clipboard
| Challenge: | A large number of reading comprehension (RC) datasets have been created, but little research has been done on whether they generalize to one another and the extent to which existing datasets can be leveraged for improving performance on new ones. |
| Approach: | They propose a BERT-based reading comprehension model that can be trained on multiple RC datasets. |
| Outcome: | The proposed model can be trained on multiple RC datasets and improve performance on five RC data. |
Cut to the Chase: A Context Zoom-in Network for Reading Comprehension (D18-1)
Copied to clipboard
| Challenge: | Recent deep-learning based models suffer from reasoning over long documents and do not trivially generalize to cases where the answer is not present as a span. |
| Approach: | They propose a novel context zoom-in network (ConZNet) that can skip through irrelevant parts of a document and generate an answer using only the relevant regions of text. |
| Outcome: | The proposed architecture outperforms state-of-the-art results by 12.62% (ROUGE-L) relative improvement on the recently proposed and challenging RC dataset ‘NarrativeQA’. |
R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies have revealed that reading comprehension (RC) systems learn to exploit annotation artifacts and other biases in current datasets. |
| Approach: | They propose a task that requires giving answers and derivations to evaluate RC systems' internal reasoning. |
| Outcome: | The proposed framework annotates 4.6k questions with 3 reference derivations and shows that it is reliable and compares with existing benchmarks. |
What do Models Learn from Question Answering Datasets? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models have outperformed humans on question answering datasets, but they have yet to outperform humans on the task of question answering itself. |
| Approach: | They evaluate BERT-based question answering models on their generalizability to out-of-domain examples, responses to missing or incorrect data, and ability to handle question variations. |
| Outcome: | The proposed models outperform human baselines on the widely-used SQuAD 1.1 and SQu AD 2.0 datasets. |
IIRC: A Dataset of Incomplete Information Reading Comprehension Questions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing reading comprehension tasks focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system’s performance at identifying a potential lack of sufficient information and locating sources for that information. |
| Approach: | They propose to use a dataset with 13K questions over paragraphs from English Wikipedia that provide only partial information to answer them, with the missing information occurring in one or more linked documents. |
| Outcome: | The proposed model achieves 31.1% F1 on the reading comprehension task, while estimated human performance is 88.4%. |
Generalizing Question Answering System with Pre-trained Language Model Fine-tuning (D19-58)
Copied to clipboard
| Challenge: | Existing methods focus on improving in-domain performance, leaving open the question of how they can generalize to out-of-domain and unseen RC tasks. |
| Approach: | They propose a multi-task learning framework that learns the shared representation across different tasks and builds on a large pre-trained language model and fine-tuned on multiple RC datasets. |
| Outcome: | The proposed framework improves the BERT-Large baseline by 8.39 and 7.22 respectively. |
NUT-RC: Noisy User-generated Text-oriented Reading Comprehension (2020.coling-main)
Copied to clipboard
| Challenge: | Existing RC models focus on extractive or generative, but ignore integration of them. |
| Approach: | They propose a noisy user-generated text-oriented RC model that integrates extractive and generative RC models by a multi-task learning mechanism and an answer selection module. |
| Outcome: | The proposed model outperforms state-of-the-art models on Twitter. |
TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Current machine reading comprehension benchmarks have no questions that test temporal phenomena . a new study studies reading comprehension for temporal relations . |
| Approach: | They propose a reading comprehension benchmark built on news snippets and 21k human-generated questions querying temporal relationships. |
| Outcome: | The new reading comprehension benchmark TORQUE achieves an exact-match score of 51% on the test set . the benchmark is built on 3.2k news snippets with 21k human-generated questions . |
On Making Reading Comprehension More Comprehensive (D19-58)
Copied to clipboard
| Challenge: | Getting machines to "understand" text is a vast and long-standing problem, made more challenging by the fact that it is not even clear what it means to understand text. |
| Approach: | They propose a question-based approach to machine reading comprehension that uses a natural language question to test a system's comprehension of a passage of text. |
| Outcome: | The proposed questions have surface cues or other biases that allow a model to shortcut the intended reasoning process. |
Towards Interpreting BERT for Reading Comprehension Based QA (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models such as ELMO and XLNet have achieved state-of-the-art performance on various NLP tasks. |
| Approach: | They propose to define a layer’s role or functionality using Integrated Gradients and perform preliminary analysis across all layers. |
| Outcome: | The proposed model performs better than existing models on RCQA and ELMO, but it lacks the human-level performance needed to perform the task. |