Causal Intervention for Mitigating Name Bias in Machine Reading Comprehension (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing MRC models may overuse name information to make predictions, causing name bias . |
| Approach: | They propose a Causal Interventional paradigm for MRC to mitigate name bias by analyzing pre-trained knowledge and context representations. |
| Outcome: | The proposed model is robust to names and performs competitively on the original SQuAD. |
Similar Papers
A Causal View of Entity Bias in (Large) Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Entity bias affects pretrained (large) language models, causing them to rely on (biased) parametric knowledge to make unfaithful predictions. |
| Approach: | They propose a structured causal model whose parameters are easier to estimate . they propose to perturb the original entity with neighboring entities . |
| Outcome: | The proposed model reduces biasing information pertaining to the original entity while still preserving sufficient semantic information from similar entities. |
On the Robustness of Reading Comprehension Models to Entity Renaming (2022.naacl-main)
Copied to clipboard
| Challenge: | SpanBERT model is more robust than RoBERTa, despite having similar accuracy on unperturbed test data. |
| Approach: | They propose a pipeline to replace entity names with names from a variety of sources. |
| Outcome: | The proposed model performs worse when entities are renamed, the authors show . SpanBERT, which is pretrained with span-level masking, is more robust than RoBERTa . |
Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models (2021.acl-short)
Copied to clipboard
| Challenge: | Pre-trained language models have achieved human-level performance on many Machine Reading Comprehension (MRC) tasks, but it remains unclear whether these models truly understand language or answer questions by exploiting statistical biases in datasets. |
| Approach: | They propose a method to attack MRC models by exposing statistical biases in a RACE dataset and propose an augmented training method that can greatly reduce models’ statistical bias. |
| Outcome: | The proposed method can reduce models’ statistical biases from human-level performance to chance-level. |
Improving the Robustness of Deep Reading Comprehension Models by Leveraging Syntax Prior (D19-58)
Copied to clipboard
| Challenge: | Recent studies indicate that the current machine reading comprehension systems suffer from weak robustness against adversarial samples. |
| Approach: | They propose to take sentence syntax as the leverage in the answer predicting process and exploit the syntactic elements of a question to improve the generalization and robustness of MRC models. |
| Outcome: | The proposed method improves generalization and robustness against adversarial samples, with performance well-maintained. |
Machine Reading Comprehension using Case-based Reasoning (2023.findings-emnlp)
Copied to clipboard
Dung Thai, Dhruv Agarwal, Mudit Chaudhary, Wenlong Zhao, Rajarshi Das, Jay-Yoon Lee, Hannaneh Hajishirzi, Manzil Zaheer, Andrew McCallum
| Challenge: | Current state-of-the-art machine readers do not support case-based reasoning . |
| Approach: | They propose a method that extracts a set of similar cases from a nonparametric memory and then predicts an answer by selecting the span in the test context that is most similar to the contextualized representations of answers. |
| Outcome: | The proposed method outperforms baselines on NaturalQuestions and NewsQA by 11.5 and 8.4 EM. |
Adversarial Domain Adaptation for Machine Reading Comprehension (D19-1)
Copied to clipboard
| Challenge: | Existing models for machine reading comprehension rely on large amounts of human-annotated in-domain data. |
| Approach: | They propose an unsupervised domain adaptation framework for Machine Reading Comprehension where the source domain has a large amount of labeled data, while only unlabeled passages are available in the target domain. |
| Outcome: | The proposed framework can be generalizable to different MRC models and datasets and can be extended to semi-supervised learning. |
Benchmarking Machine Reading Comprehension: A Psychological Perspective (2021.eacl-main)
Copied to clipboard
| Challenge: | MRC is a task that tests the ability of a machine to read and understand unstructured text. |
| Approach: | They propose a theoretical basis for the design of MRC datasets based on psychology and psychometrics and propose shortcut-proof questions and explanations as a part of the task design. |
| Outcome: | The proposed datasets should evaluate the model's ability to understand context-dependent situations and ensure substantive validity by shortcut-proof questions and explanation as a part of the task design. |
Improving Machine Reading Comprehension with General Reading Strategies (N19-1)
Copied to clipboard
| Challenge: | Recent studies have shown that reading strategies improve comprehension levels for readers lacking adequate prior knowledge. |
| Approach: | They propose three general strategies to improve machine reading comprehension (MRC) by fine-tuning a pre-trained model with strategies and a target task. |
| Outcome: | The proposed models improve non-extractive machine reading comprehension (MRC) on the largest general domain multiple-choice dataset RACE. |
Aligning as Debiasing: Causality-Aware Alignment via Reinforcement Learning with Interventional Feedback (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to reduce LLMs' biased outputs rely on reward signals from current model outputs without considering the source of biases. |
| Approach: | They propose to leverage the reward model in RL alignment as an instrumental variable to perform causal intervention on LLMs. |
| Outcome: | The proposed method reduces biases by using human feedback to fine tune LLMs to human values. |
Enhancing Multiple-choice Machine Reading Comprehension by Punishing Illogical Interpretations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Multiple-choice MRC is one of the most studied tasks in MRC due to the convenience of evaluation and the flexibility of answer format. |
| Approach: | They propose to use multiple-choice MRC to explain a trained model and reveal how it arrives at the prediction by punishing illogical attributions. |
| Outcome: | The proposed method improves model performance without external information and model structure change without any external information. |