Towards Robust Extractive Question Answering Models: Rethinking the Training Methodology (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models lack robustness against distribution shifts and adversarial attacks when training on unanswerable questions in EQA datasets. |
| Approach: | They propose a novel loss function for the EQA problem to improve the robustness of extractive question answering models by adding adversarial questions to a crowdsourcing process. |
| Outcome: | The proposed method maintains in-domain performance while improving on out-of-domain datasets. |
Similar Papers
Robust Machine Comprehension Models via Adversarial Training (N18-2)
Copied to clipboard
| Challenge: | Existing models for the Stanford Question Answering Dataset suffer from a 50% decrease in F1 score during adversarial evaluation based on AddSent. |
| Approach: | They propose an alternative adversary-generation algorithm, AddSentDiverse, that significantly increases the variance within the adversarial training data by providing effective examples that punish the model for making certain superficial assumptions. |
| Outcome: | The proposed algorithm can achieve a 36.5% increase in F1 score while maintaining performance on the regular SQuAD task. |
RobustQA: Benchmarking the Robustness of Domain Adaptation for Open-Domain Question Answering (2023.findings-acl)
Copied to clipboard
Rujun Han, Peng Qi, Yuhao Zhang, Lan Liu, Juliette Burger, William Yang Wang, Zhiheng Huang, Bing Xiang, Dan Roth
| Challenge: | Existing ODQA datasets consist mainly of Wikipedia corpus, and are insufficient to study models’ generalizability across diverse domains. |
| Approach: | They propose a benchmark to evaluate ODQA's domain robustness using Wikipedia corpus . they annotate QA pairs in retrieval datasets with rigorous quality control . |
| Outcome: | The proposed benchmark improves model performance on annotated QA pairs in retrieval datasets with rigorous quality control. |
Exploring The Landscape of Distributional Robustness for Question Answering Models (2022.findings-emnlp)
Copied to clipboard
Anas Awadalla, Mitchell Wortsman, Gabriel Ilharco, Sewon Min, Ian Magnusson, Hannaneh Hajishirzi, Ludwig Schmidt
| Challenge: | Existing methods for predicting distributional robustness fail to generalize reliably in a variety of test conditions. |
| Approach: | They conduct a large empirical evaluation to investigate the landscape of distributional robustness in question answering. |
| Outcome: | The proposed methods are more robust to distribution shifts than fully fine-tuned models, and few-shot prompt models exhibit better robustness than few- shot prompt models. |
End-to-End Training of Neural Retrievers for Open-Domain Question Answering (2021.acl-long)
Copied to clipboard
Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton, Bryan Catanzaro
| Challenge: | Recent work on training neural retrievers for open-domain question answering (OpenQA) has employed both supervised and unsupervised methods. |
| Approach: | They propose an approach of unsupervised pre-training with the Inverse Cloze Task and masked salient spans followed by supervised finetuning using question-context pairs. |
| Outcome: | The proposed approach outperforms models like REALM and RAG in retrieval accuracy and answer extraction. |
UnitedQA: A Hybrid Approach for Open Domain Question Answering (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work on open-domain question answering focuses on either extractive or generative readers exclusively. |
| Approach: | They propose a hybrid approach to extractive and generative readers that leverages both models. |
| Outcome: | The proposed approach outperforms state-of-the-art models on NaturalQuestions and TriviaQA respectively. |
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | a new approach to generate adversarial data is needed to improve question answering models . crowdworkers can fool a model only 8.8% of the time, compared to 17.6% for a trained model without synthetic data. |
| Approach: | They develop a pipeline that generates questions and then filters or labels them to improve quality. |
| Outcome: | The proposed approach improves state-of-the-art on a human-written adversarial dataset by 3.7F1 and improves model generalisation on nine of the twelve MRQA datasets. |
RobustQA: A Framework for Adversarial Text Generation Analysis on Question Answering Systems (2023.emnlp-demo)
Copied to clipboard
Yasaman Boreshban, Seyed Morteza Mirbostani, Seyedeh Fatemeh Ahmadi, Gita Shojaee, Fatemeh Kamani, Gholamreza Ghassem-Sani, Seyed Abolghasem Mirroshandel
| Challenge: | Question answering (QA) systems have reached human-level accuracy, but they are not robust enough and vulnerable to adversarial examples. |
| Approach: | They modified the attack algorithms widely used in text classification to fit them for QA systems. |
| Outcome: | The proposed framework is the first open-source toolkit for investigating textual adversarial attacks in QA systems. |
On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study (2021.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that adversarial data collection (ADC) models perform better on other adversarially collected data but are liable under plausible domain shifts. |
| Approach: | They conduct a large-scale controlled study on question answering by assigning workers at random to compose questions either adversarially (with a model in the loop) or in the standard fashion (without a modeling). |
| Outcome: | The proposed model performs better on other adversarial datasets but worse on diverse collection of out-of-domain evaluation sets. |
Domain-agnostic Question-Answering with Adversarial Training (D19-58)
Copied to clipboard
| Challenge: | Adapting models to new domain without finetuning is a challenging problem in deep learning. |
| Approach: | They propose an adversarial training framework for domain generalization in Question Answering task using a conventional QA model and a discriminator. |
| Outcome: | The proposed model outperforms the baseline model on Question Answering (QA) task. |
Learning Invariant Representation Improves Robustness for MRC Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to improve machine reading comprehension models are vulnerable and not robust to adversarial examples. |
| Approach: | They propose to construct positive example pairs which have same answer by augmentation and then introduce stability and contrastive loss to improve invariance of representation. |
| Outcome: | The proposed approach boosts the robustness of QA models across different tasks and attack sets significantly and consistently. |