| Challenge: | Recent research aims to facilitate training and evaluation on several reading comprehension datasets at the same time. |
| Approach: | They propose an evaluation server that reports performance on seven diverse reading comprehension datasets and includes synthetic augmentations to test models' ability to handle out-of-domain questions. |
| Outcome: | The evaluation server performs on seven reading comprehension datasets, and collects and includes synthetic augmentations for these datasets to test models' ability to handle out-of-domain questions. |
Similar Papers
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (P19-1)
Copied to clipboard
| Challenge: | A large number of reading comprehension (RC) datasets have been created, but little research has been done on whether they generalize to one another and the extent to which existing datasets can be leveraged for improving performance on new ones. |
| Approach: | They propose a BERT-based reading comprehension model that can be trained on multiple RC datasets. |
| Outcome: | The proposed model can be trained on multiple RC datasets and improve performance on five RC data. |
Dataset for the First Evaluation on Chinese Machine Reading Comprehension (L18-1)
Copied to clipboard
| Challenge: | Existing reading comprehension datasets are mostly in English . |
| Approach: | They propose a Chinese reading comprehension dataset to add diversity to existing reading comprehension data . proposed dataset contains cloze-style reading comprehension and user query reading comprehension . |
| Outcome: | The proposed dataset is based on a Chinese reading comprehension dataset . it includes two types of cloze-style and user query reading comprehension . the proposed dataset hosted the 1st Evaluation on Chinese Machine Reading Comprehension (CMRC-2017) |
On Making Reading Comprehension More Comprehensive (D19-58)
Copied to clipboard
| Challenge: | Getting machines to "understand" text is a vast and long-standing problem, made more challenging by the fact that it is not even clear what it means to understand text. |
| Approach: | They propose a question-based approach to machine reading comprehension that uses a natural language question to test a system's comprehension of a passage of text. |
| Outcome: | The proposed questions have surface cues or other biases that allow a model to shortcut the intended reasoning process. |
Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset (2020.acl-main)
Copied to clipboard
| Challenge: | Medical professionals often query over clinical notes to find information that can support their decision making. |
| Approach: | They propose to use expert-annotated question templates and existing i2b2 annotations to create emrQA, the first large-scale dataset for question answering based on clinical notes. |
| Outcome: | The proposed system can answer clinical questions without using domain knowledge. |
Interpretable Multi-dataset Evaluation for Named Entity Recognition (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation methods for named entity recognition tasks are difficult to interpret . authors present a general methodology for interpretable evaluation for named entities . |
| Approach: | They propose a general methodology for interpretable evaluation for named entity recognition task. |
| Outcome: | The proposed evaluation method enables researchers to interpret differences in models and datasets . it makes it easy for future researchers to run similar analyses and drive progress in this area . |
How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks (D18-1)
Copied to clipboard
| Challenge: | Recent research addresses reading comprehension, where examples consist of (question, passage, answer) tuples. |
| Approach: | They establish sensible baselines for bAbI, SQuAD, CBT, CNN and Who-did-What datasets and compare them to their previous work. |
| Outcome: | The proposed models perform on 14 out of 20 bAbI, SQuAD, CBT, CNN and Who-did-What datasets. |
SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers (2024.emnlp-main)
Copied to clipboard
| Challenge: | SciDQA is a dataset for question-answering that challenges language models to deeply understand scientific articles. |
| Approach: | They propose a new dataset for reading comprehension that challenges language models to deeply understand scientific articles consisting of 2,937 QA pairs. |
| Outcome: | The SciDQA dataset is based on 2,937 QA pairs and decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering. |
English Machine Reading Comprehension Datasets: A Survey (2021.emnlp-main)
Copied to clipboard
| Challenge: | a survey of English Machine Reading Comprehension datasets is carried out . the aim is to provide a concise yet informative overview of the landscape . |
| Approach: | They survey 60 English Machine Reading Comprehension datasets to provide a resource for other researchers interested in this problem. |
| Outcome: | The proposed survey covers 60 English MRC datasets with a view to providing a resource for other researchers interested in the problem. |
Automatic learner summary assessment for reading comprehension (N19-1)
Copied to clipboard
| Challenge: | Summarization is a well-established method of measuring reading proficiency in traditional English as a second or other language assessments. |
| Approach: | They propose three approaches to automatically assess learner summary for evaluating non-native reading comprehension using a summarization task and a long-term memory model. |
| Outcome: | The proposed models outperform traditional methods and produce quality assessments close to professional examiners. |
Benchmarking Machine Reading Comprehension: A Psychological Perspective (2021.eacl-main)
Copied to clipboard
| Challenge: | MRC is a task that tests the ability of a machine to read and understand unstructured text. |
| Approach: | They propose a theoretical basis for the design of MRC datasets based on psychology and psychometrics and propose shortcut-proof questions and explanations as a part of the task design. |
| Outcome: | The proposed datasets should evaluate the model's ability to understand context-dependent situations and ensure substantive validity by shortcut-proof questions and explanation as a part of the task design. |