| Challenge: | Existing datasets focus on answerable questions or use automatically generated unanswerable questions that are easy to identify. |
| Approach: | They propose a dataset that combines the Stanford Question Answering Dataset with 50,000 unanswerable questions written by crowdworkers to look similar to answerable ones. |
| Outcome: | The proposed dataset looks similar to answerable questions on crowd-written questions . strong neural system that gets 86% F1 on SQuAD achieves only 66% F1. |
Similar Papers
SQuAD2-CR: Semi-supervised Annotation for Cause and Rationales for Unanswerability in SQuAD 2.0 (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing models are brittle for adversarial perturbed questions, causing uncertainty . a dataset with annotations on unanswerable questions is not available to solve this problem . |
| Approach: | They use crowdsourced annotations to annotate unanswerable questions . they also annotated which part of the question causes unanswered questions a . |
| Outcome: | The proposed dataset can be used to improve model interpretation, authors say . they find that existing models are brittle for adversarial perturbed questions . |
Do We Know What We Don’t Know? Studying Unanswerable Questions beyond SQuAD 2.0 (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has attempted to improve extractive QA performance by enriching the dataset with unanswerable questions. |
| Approach: | They build an out-of-domain corpus of competitive and non-competitive questions . they compare the results with the results of the Recognizing Textual Entailments task . |
| Outcome: | The proposed model fails even in the case of simpler questions . the proposed model can be used to address more realistic situations in reading comprehension . |
Learning to Ask Unanswerable Questions for Machine Reading Comprehension (P19-1)
Copied to clipboard
| Challenge: | Existing models for extractive reading comprehension are not good at deciding whether no answer is presented in the context. |
| Approach: | They propose a data augmentation technique by automatically generating relevant unanswerable questions according to an answerable question paired with its corresponding paragraph that contains the answer. |
| Outcome: | The proposed model performs better on the SQuAD 2.0 dataset than the baseline model and the BERT-large model. |
The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Pretrained language models have achieved super-human performances on many Machine Reading Comprehension (MRC) benchmarks. |
| Approach: | They propose to fine-tune three state-of-the-art language models on SQuAD 1.1 or SQu AD 2.0 and then evaluate their robustness under adversarial attacks. |
| Outcome: | The proposed model is able to perform better under adversarial attacks than model fine-tuned on SQuAD 1.1 or 2.0. |
A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC (N19-1)
Copied to clipboard
| Challenge: | In response to this development, there have been a flurry of new datasets for question answering. |
| Approach: | They propose to use SQuAD 2.0, QuAC, and CoQA to provide question answering on textual data. |
| Outcome: | The proposed datasets provide complementary coverage of the first two aspects, but weak coverage of third. |
I Could’ve Asked That: Reformulating Unanswerable Questions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language models do not assist users in reformulating unanswerable questions . a recent study found that the models failed to reformulate questions based on assumptions that conflict with or cannot be verified with the information available in documents. |
| Approach: | They evaluate open-source and proprietary LLMs on couldAsk to evaluate their performance . they found that GPT-4 and Llama2-7B successfully reformulate questions only 26% and 12% of the time . |
| Outcome: | The proposed model successfully reformulates questions only 26% and 12% of the time . the proposed model is not able to reformulate questions, but it can be improved . |
What do Models Learn from Question Answering Datasets? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models have outperformed humans on question answering datasets, but they have yet to outperform humans on the task of question answering itself. |
| Approach: | They evaluate BERT-based question answering models on their generalizability to out-of-domain examples, responses to missing or incorrect data, and ability to handle question variations. |
| Outcome: | The proposed models outperform human baselines on the widely-used SQuAD 1.1 and SQu AD 2.0 datasets. |
Contextualized Word Representations for Reading Comprehension (N18-2)
Copied to clipboard
| Challenge: | Reading comprehension (RC) is a high-level task in natural language understanding that requires reading a document and answering questions about its content. |
| Approach: | They propose to provide a standard neural network for reading a document and answering a question about its content. |
| Outcome: | The proposed model improves on the competitive SQuAD dataset by providing rich contextualized word representations and allowing it to choose between context-dependent and context-independent representations. |
IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehension (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing MRC datasets in Indonesian are inadequate because of the small size and limited question types. |
| Approach: | They propose to combine automatic and manual unanswerable question generation to minimize the cost of manual dataset construction while maintaining the dataset quality. |
| Outcome: | The proposed dataset significantly improves the performance of Indonesian MRC models, showing a large improvement for unanswerable questions. |
FQuAD2.0: French Question Answering and Learning When You Don’t Know (2022.lrec-1)
Copied to clipboard
| Challenge: | Question Answering, including Reading Comprehension, has seen significant scientific breakthroughs over the past few years . but most of these breakthroughs are centered on the English language . |
| Approach: | They propose a dataset to train Question Answering models in the French language . they extend the dataset to 17,000+ unanswerable questions annotated adversarially . |
| Outcome: | The proposed dataset makes it possible to train French Question Answering models with the ability to distinguish unanswerable questions from answerable ones. |