Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning (D19-1)
Copied to clipboard
| Challenge: | Existing reading comprehension benchmarks do not contain complex coreferential phenomena . obtaining questions focused on such phenomena is difficult because of lexical cues . |
| Approach: | They propose to use a crowdsourced dataset to examine the ability of models to resolve coreference among entities in Wikipedia paragraphs. |
| Outcome: | The proposed model performs significantly worse than humans on the reading comprehension benchmark . paragraphs and other longer texts typically make multiple references to the same entities . |
Similar Papers
Coreference Reasoning in Machine Reading Comprehension (2021.acl-long)
Copied to clipboard
| Challenge: | Existing datasets for machine reading comprehension do not reflect the natural distribution and, consequently, the challenges of coreference reasoning. |
| Approach: | They propose to use existing coreference resolution datasets to train machine reading comprehension models to better reflect the natural distribution and, consequently, the challenges of coreference reasoning. |
| Outcome: | The proposed method improves the performance of state-of-the-art models on a set of coreference-related datasets. |
Tracing Origins: Coreference-aware Machine Reading Comprehension (2022.acl-long)
Copied to clipboard
| Challenge: | a recent study has enriched pre-trained language models with syntactic, semantic and other linguistic information to improve their performance. |
| Approach: | They use a pre-trained language model to leverage coreference information to enhance word embeddings . they use additional encoder layers to focus on coreference mentions or a relational graph convolutional network to model the coreference relations. |
| Outcome: | The proposed model imitates the human reading process and leverages coreference information to enhance word embeddings. |
How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks (D18-1)
Copied to clipboard
| Challenge: | Recent research addresses reading comprehension, where examples consist of (question, passage, answer) tuples. |
| Approach: | They establish sensible baselines for bAbI, SQuAD, CBT, CNN and Who-did-What datasets and compare them to their previous work. |
| Outcome: | The proposed models perform on 14 out of 20 bAbI, SQuAD, CBT, CNN and Who-did-What datasets. |
IIRC: A Dataset of Incomplete Information Reading Comprehension Questions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing reading comprehension tasks focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system’s performance at identifying a potential lack of sufficient information and locating sources for that information. |
| Approach: | They propose to use a dataset with 13K questions over paragraphs from English Wikipedia that provide only partial information to answer them, with the missing information occurring in one or more linked documents. |
| Outcome: | The proposed model achieves 31.1% F1 on the reading comprehension task, while estimated human performance is 88.4%. |
What Makes Reading Comprehension Questions Difficult? (2022.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that natural language understanding benchmarks are not able to measure future progress . a crowdsourcing approach is needed to collect diverse examples without sacrificing diversity or coverage. |
| Approach: | They crowdsource multiple-choice reading comprehension questions for passages from seven sources . they find passage source, length, and readability measures do not significantly affect question difficulty . |
| Outcome: | The results show that passage source, length, and readability measures do not significantly affect question difficulty. |
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (N18-1)
Copied to clipboard
| Challenge: | Using a dataset of 6,500+ questions, we found that human solvers achieved an F1-score of 88.1%. |
| Approach: | They propose a reading comprehension challenge in which questions can only be answered by taking into account information from multiple sentences. |
| Outcome: | The proposed reading comprehension challenge is based on a reading comprehension dataset with 6,500+ questions and 1000+ paragraphs across 7 domains. |
English Machine Reading Comprehension Datasets: A Survey (2021.emnlp-main)
Copied to clipboard
| Challenge: | a survey of English Machine Reading Comprehension datasets is carried out . the aim is to provide a concise yet informative overview of the landscape . |
| Approach: | They survey 60 English Machine Reading Comprehension datasets to provide a resource for other researchers interested in this problem. |
| Outcome: | The proposed survey covers 60 English MRC datasets with a view to providing a resource for other researchers interested in the problem. |
On Making Reading Comprehension More Comprehensive (D19-58)
Copied to clipboard
| Challenge: | Getting machines to "understand" text is a vast and long-standing problem, made more challenging by the fact that it is not even clear what it means to understand text. |
| Approach: | They propose a question-based approach to machine reading comprehension that uses a natural language question to test a system's comprehension of a passage of text. |
| Outcome: | The proposed questions have surface cues or other biases that allow a model to shortcut the intended reasoning process. |
Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension (2020.tacl-1)
Copied to clipboard
| Challenge: | Innovations in annotation methodologies have been a catalyst for Reading Comprehension (RC) datasets and models. |
| Approach: | They propose to use a model-in-the-annotation-loop approach to train adversarial models in three different settings to explore reproducibility of the adversarial effect, transfer from data collected with varying model- in-the loop strengths, and generalization to data collected without a modeling model. |
| Outcome: | The proposed approach achieves 39.9F1 on questions it cannot answer when trained on SQUAD, but lower than when trained using RoBERTa itself (41.0F1). |
Probing Neural Network Comprehension of Natural Language Arguments (P19-1)
Copied to clipboard
| Challenge: | Argument Reasoning Comprehension Task (ARCT) focuses on inferences, not just discovering warrants. |
| Approach: | They propose to build an adversarial dataset on which all models achieve random accuracy. |
| Outcome: | The proposed dataset provides a more robust assessment of argument comprehension and should be adopted as the standard in future work. |