| Challenge: | Getting machines to "understand" text is a vast and long-standing problem, made more challenging by the fact that it is not even clear what it means to understand text. |
| Approach: | They propose a question-based approach to machine reading comprehension that uses a natural language question to test a system's comprehension of a passage of text. |
| Outcome: | The proposed questions have surface cues or other biases that allow a model to shortcut the intended reasoning process. |
Similar Papers
To Test Machine Comprehension, Start by Defining Comprehension (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to machine reading comprehension do not adequately define comprehension, authors argue . authors argue that existing systems are not up to the task of narrative understanding as they define it . |
| Approach: | They propose a detailed definition of comprehension for short narratives . they argue existing systems are not up to the task of narrative understanding . |
| Outcome: | The proposed task definitions suggest existing systems are not up to the task of narrative understanding as they define it. |
How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks (D18-1)
Copied to clipboard
| Challenge: | Recent research addresses reading comprehension, where examples consist of (question, passage, answer) tuples. |
| Approach: | They establish sensible baselines for bAbI, SQuAD, CBT, CNN and Who-did-What datasets and compare them to their previous work. |
| Outcome: | The proposed models perform on 14 out of 20 bAbI, SQuAD, CBT, CNN and Who-did-What datasets. |
Why Machine Reading Comprehension Models Learn Shortcuts? (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that many MRC models learn shortcuts to outwit benchmarks, but the performance is unsatisfactory in real-world applications. |
| Approach: | They propose to use shortcut questions to analyze learning difficulty of MRC models . they propose to analyze the learning difficulty regarding shortcut and challenging questions . |
| Outcome: | The proposed methods show that a large proportion of shortcut questions in training data make models rely on shortcut tricks excessively. |
What Makes Reading Comprehension Questions Difficult? (2022.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that natural language understanding benchmarks are not able to measure future progress . a crowdsourcing approach is needed to collect diverse examples without sacrificing diversity or coverage. |
| Approach: | They crowdsource multiple-choice reading comprehension questions for passages from seven sources . they find passage source, length, and readability measures do not significantly affect question difficulty . |
| Outcome: | The results show that passage source, length, and readability measures do not significantly affect question difficulty. |
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (N18-1)
Copied to clipboard
| Challenge: | Using a dataset of 6,500+ questions, we found that human solvers achieved an F1-score of 88.1%. |
| Approach: | They propose a reading comprehension challenge in which questions can only be answered by taking into account information from multiple sentences. |
| Outcome: | The proposed reading comprehension challenge is based on a reading comprehension dataset with 6,500+ questions and 1000+ paragraphs across 7 domains. |
English Machine Reading Comprehension Datasets: A Survey (2021.emnlp-main)
Copied to clipboard
| Challenge: | a survey of English Machine Reading Comprehension datasets is carried out . the aim is to provide a concise yet informative overview of the landscape . |
| Approach: | They survey 60 English Machine Reading Comprehension datasets to provide a resource for other researchers interested in this problem. |
| Outcome: | The proposed survey covers 60 English MRC datasets with a view to providing a resource for other researchers interested in the problem. |
Machine Reading Comprehension using Case-based Reasoning (2023.findings-emnlp)
Copied to clipboard
Dung Thai, Dhruv Agarwal, Mudit Chaudhary, Wenlong Zhao, Rajarshi Das, Jay-Yoon Lee, Hannaneh Hajishirzi, Manzil Zaheer, Andrew McCallum
| Challenge: | Current state-of-the-art machine readers do not support case-based reasoning . |
| Approach: | They propose a method that extracts a set of similar cases from a nonparametric memory and then predicts an answer by selecting the span in the test context that is most similar to the contextualized representations of answers. |
| Outcome: | The proposed method outperforms baselines on NaturalQuestions and NewsQA by 11.5 and 8.4 EM. |
What Makes Reading Comprehension Questions Easier? (D18-1)
Copied to clipboard
| Challenge: | Recent studies have shown that questions require a deeper understanding of language to answer beyond using superficial cues. |
| Approach: | They propose to use simple heuristics to split MRC datasets into easy and hard subsets and manually annotate questions from each subset with validity and reasoning skills to investigate which skills explain the difference between easy and harder questions. |
| Outcome: | The proposed model performs better for hard and easy questions than for easy questions. |
Knowing More About Questions Can Help: Improving Calibration in Question Answering (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing work on calibration focuses on model confidence, such as the max probability of the predicted class. |
| Approach: | They propose a calibration method which estimates whether model correctly predicts answer for each question. |
| Outcome: | The proposed calibration method achieves 5-10% gains on reading comprehension benchmarks. |
Automatic learner summary assessment for reading comprehension (N19-1)
Copied to clipboard
| Challenge: | Summarization is a well-established method of measuring reading proficiency in traditional English as a second or other language assessments. |
| Approach: | They propose three approaches to automatically assess learner summary for evaluating non-native reading comprehension using a summarization task and a long-term memory model. |
| Outcome: | The proposed models outperform traditional methods and produce quality assessments close to professional examiners. |