The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Pretrained language models have achieved super-human performances on many Machine Reading Comprehension (MRC) benchmarks. |
| Approach: | They propose to fine-tune three state-of-the-art language models on SQuAD 1.1 or SQu AD 2.0 and then evaluate their robustness under adversarial attacks. |
| Outcome: | The proposed model is able to perform better under adversarial attacks than model fine-tuned on SQuAD 1.1 or 2.0. |
Similar Papers
Evaluating Neural Model Robustness for Machine Comprehension (2021.eacl-main)
Copied to clipboard
| Challenge: | evaluating model robustness to adversarial attacks can provide deeper understanding of how deep neural networks work and what kind of linguistic information is actually captured by neural networks. |
| Approach: | They propose a method for strategic sentence-level perturbations to evaluate model robustness to adversarial attacks using character and word perturbations. |
| Outcome: | The proposed model improves model performance during adversarial attacks by using ensembles and predicts errors in adversarials. |
Know What You Don’t Know: Unanswerable Questions for SQuAD (P18-2)
Copied to clipboard
| Challenge: | Existing datasets focus on answerable questions or use automatically generated unanswerable questions that are easy to identify. |
| Approach: | They propose a dataset that combines the Stanford Question Answering Dataset with 50,000 unanswerable questions written by crowdworkers to look similar to answerable ones. |
| Outcome: | The proposed dataset looks similar to answerable questions on crowd-written questions . strong neural system that gets 86% F1 on SQuAD achieves only 66% F1. |
Robust Machine Comprehension Models via Adversarial Training (N18-2)
Copied to clipboard
| Challenge: | Existing models for the Stanford Question Answering Dataset suffer from a 50% decrease in F1 score during adversarial evaluation based on AddSent. |
| Approach: | They propose an alternative adversary-generation algorithm, AddSentDiverse, that significantly increases the variance within the adversarial training data by providing effective examples that punish the model for making certain superficial assumptions. |
| Outcome: | The proposed algorithm can achieve a 36.5% increase in F1 score while maintaining performance on the regular SQuAD task. |
SQuAD2-CR: Semi-supervised Annotation for Cause and Rationales for Unanswerability in SQuAD 2.0 (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing models are brittle for adversarial perturbed questions, causing uncertainty . a dataset with annotations on unanswerable questions is not available to solve this problem . |
| Approach: | They use crowdsourced annotations to annotate unanswerable questions . they also annotated which part of the question causes unanswered questions a . |
| Outcome: | The proposed dataset can be used to improve model interpretation, authors say . they find that existing models are brittle for adversarial perturbed questions . |
Benchmarking Robustness of Machine Reading Comprehension Models (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks only evaluate models' robustness under test-time perturbations or adversarial attacks. |
| Approach: | They propose a model-agnostic benchmark to evaluate models' robustness under adversarial attacks. |
| Outcome: | The proposed model-agnostic benchmark evaluates models under four different types of adversarial attacks. |
On the Robustness of Reading Comprehension Models to Entity Renaming (2022.naacl-main)
Copied to clipboard
| Challenge: | SpanBERT model is more robust than RoBERTa, despite having similar accuracy on unperturbed test data. |
| Approach: | They propose a pipeline to replace entity names with names from a variety of sources. |
| Outcome: | The proposed model performs worse when entities are renamed, the authors show . SpanBERT, which is pretrained with span-level masking, is more robust than RoBERTa . |
Towards Robust Extractive Question Answering Models: Rethinking the Training Methodology (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models lack robustness against distribution shifts and adversarial attacks when training on unanswerable questions in EQA datasets. |
| Approach: | They propose a novel loss function for the EQA problem to improve the robustness of extractive question answering models by adding adversarial questions to a crowdsourcing process. |
| Outcome: | The proposed method maintains in-domain performance while improving on out-of-domain datasets. |
Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models (2021.acl-short)
Copied to clipboard
| Challenge: | Pre-trained language models have achieved human-level performance on many Machine Reading Comprehension (MRC) tasks, but it remains unclear whether these models truly understand language or answer questions by exploiting statistical biases in datasets. |
| Approach: | They propose a method to attack MRC models by exposing statistical biases in a RACE dataset and propose an augmented training method that can greatly reduce models’ statistical bias. |
| Outcome: | The proposed method can reduce models’ statistical biases from human-level performance to chance-level. |
Learning Invariant Representation Improves Robustness for MRC Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to improve machine reading comprehension models are vulnerable and not robust to adversarial examples. |
| Approach: | They propose to construct positive example pairs which have same answer by augmentation and then introduce stability and contrastive loss to improve invariance of representation. |
| Outcome: | The proposed approach boosts the robustness of QA models across different tasks and attack sets significantly and consistently. |
Do We Know What We Don’t Know? Studying Unanswerable Questions beyond SQuAD 2.0 (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has attempted to improve extractive QA performance by enriching the dataset with unanswerable questions. |
| Approach: | They build an out-of-domain corpus of competitive and non-competitive questions . they compare the results with the results of the Recognizing Textual Entailments task . |
| Outcome: | The proposed model fails even in the case of simpler questions . the proposed model can be used to address more realistic situations in reading comprehension . |