Answerable or Not: Devising a Dataset for Extending Machine Reading Comprehension (C18-1)
Copied to clipboard
| Challenge: | Existing MRC algorithms assume that each question is answerable by looking at text passages, but to realize human-like language comprehension ability, a machine should be able to distinguish not-answerable questions from answerable questions. |
| Approach: | They propose a method for automatically assigning difficulty level labels to a dataset that alters an existing MRC dataset and describes the resulting dataset. |
| Outcome: | The proposed method can detect NAQs in a dataset with difficulty level labels and is valid and potentially useful in the development of advanced MRC models. |
Similar Papers
What Makes Reading Comprehension Questions Easier? (D18-1)
Copied to clipboard
| Challenge: | Recent studies have shown that questions require a deeper understanding of language to answer beyond using superficial cues. |
| Approach: | They propose to use simple heuristics to split MRC datasets into easy and hard subsets and manually annotate questions from each subset with validity and reasoning skills to investigate which skills explain the difference between easy and harder questions. |
| Outcome: | The proposed model performs better for hard and easy questions than for easy questions. |
IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehension (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing MRC datasets in Indonesian are inadequate because of the small size and limited question types. |
| Approach: | They propose to combine automatic and manual unanswerable question generation to minimize the cost of manual dataset construction while maintaining the dataset quality. |
| Outcome: | The proposed dataset significantly improves the performance of Indonesian MRC models, showing a large improvement for unanswerable questions. |
Coreference Reasoning in Machine Reading Comprehension (2021.acl-long)
Copied to clipboard
| Challenge: | Existing datasets for machine reading comprehension do not reflect the natural distribution and, consequently, the challenges of coreference reasoning. |
| Approach: | They propose to use existing coreference resolution datasets to train machine reading comprehension models to better reflect the natural distribution and, consequently, the challenges of coreference reasoning. |
| Outcome: | The proposed method improves the performance of state-of-the-art models on a set of coreference-related datasets. |
GCRC: A New Challenging MRC Dataset from Gaokao Chinese for Explainable Evaluation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing machine reading comprehension datasets lack an explainable evaluation of systems' reasoning capabilities. |
| Approach: | They propose a dataset with multi-choice questions that evaluates MRC systems' reasoning process . they use sentence-level relevant supporting facts, error reason of distractors to evaluate MRC . |
| Outcome: | The proposed dataset is more challenging and useful for identifying limitations of existing MRC systems in an explainable way. |
Self-Teaching Machines to Read and Comprehend with Large-Scale Multi-Subject Question-Answering Data (2021.findings-emnlp)
Copied to clipboard
| Challenge: | despite considerable progress, most machine reading comprehension tasks lack sufficient training data to fully exploit powerful deep neural network models. |
| Approach: | They propose to use QA data to generate more training data for machine reading comprehension tasks by crowdsourcing . they first collect a large-scale multiple-choice QA dataset for Chinese, ExamQA, and then use incomplete, yet relevant snippets returned by a web search engine as the context for each QA instance. |
| Outcome: | The proposed model improves a Chinese MRC task with +5.1% accuracy and +3.8% exact match. |
A Framework for Evaluation of Machine Reading Comprehension Gold Standards (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing literature on machine reading comprehension (MRC) data is limited on the data design of gold standards. |
| Approach: | They propose a framework to investigate linguistic features, lexical cues and ambiguity in MRC gold standards. |
| Outcome: | The proposed framework investigates the present linguistic features, required reasoning and background knowledge and factual correctness on the one hand, and the presence of lexical cues as a lower bound for the requirement of understanding on the other. |
On Making Reading Comprehension More Comprehensive (D19-58)
Copied to clipboard
| Challenge: | Getting machines to "understand" text is a vast and long-standing problem, made more challenging by the fact that it is not even clear what it means to understand text. |
| Approach: | They propose a question-based approach to machine reading comprehension that uses a natural language question to test a system's comprehension of a passage of text. |
| Outcome: | The proposed questions have surface cues or other biases that allow a model to shortcut the intended reasoning process. |
What do Models Learn from Question Answering Datasets? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models have outperformed humans on question answering datasets, but they have yet to outperform humans on the task of question answering itself. |
| Approach: | They evaluate BERT-based question answering models on their generalizability to out-of-domain examples, responses to missing or incorrect data, and ability to handle question variations. |
| Outcome: | The proposed models outperform human baselines on the widely-used SQuAD 1.1 and SQu AD 2.0 datasets. |
To Answer or Not To Answer? Improving Machine Reading Comprehension Model with Span-based Contrastive Learning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing models fail to recognize answerable questions due to subtle literal changes . MRC models are forced to perceive crucial semantic changes from slight literal differences. |
| Approach: | They propose a span-based method of Contrastive Learning which explicitly contrasts answerable questions with their answerable counterparts at the answer span level. |
| Outcome: | The proposed method improves baselines significantly and is an effective way to utilize generated questions. |
Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehension (2024.emnlp-main)
Copied to clipboard
| Challenge: | Extractive Machine Reading Comprehension (MRC) is a challenging field in the field of Natural Language Processing. |
| Approach: | They propose a Question-Attended Span Extraction module to address the limitations of generative approaches for extractive machine reading comprehension (MRC) . module significantly enhances performance of pre-trained generative language models, enabling them to surpass the extractive capabilities of advanced Large Language Models (LLMs) |
| Outcome: | The QASE module surpasses state-of-the-art models in few-shot settings. |