Hypothetical Training for Robust Machine Reading Comprehension of Tabular Context (2023.findings-acl)
Copied to clipboard
| Challenge: | Counterfactual training is expensive because of the complexity of tabular data. |
| Approach: | They propose a hypothetical training framework that uses paired examples with different hypothetical questions to supervise the direction of model gradient towards the counterfactual answer change. |
| Outcome: | The proposed framework improves on tabular MRC datasets. |
Similar Papers
Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning (2022.acl-long)
Copied to clipboard
| Challenge: | Existing NDR models suffer from large performance drop on hypothetical questions, e.g., “what the annualized rate of return would be if the revenue in 2020 was doubled”. |
| Approach: | They propose a learning to imagine module which can be seamlessly incorporated into NDR models to perform the imagination of unseen counterfactual. |
| Outcome: | The proposed model can perform the imagination of unseen counterfactuals on hypothetical questions. |
Enhancing Tabular Reasoning with Pattern Exploiting Training (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing methods based on pre-trained language models have shown superior performance over tabular tasks despite showing inherent problems such as not using the right evidence and inconsistent predictions across inputs. |
| Approach: | They utilize Pattern-Exploiting Training (PET) on pre-trained language models to strengthen tabular reasoning models’ pre-existing knowledge and reasoning abilities. |
| Outcome: | The proposed model exhibits superior understanding of knowledge facts and tabular reasoning compared to baseline models. |
Realistic Data Augmentation Framework for Enhancing Tabular Reasoning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to constructing training data for Natural Language Inference (NLI) tasks are expensive and time consuming. |
| Approach: | They propose a semi-automated framework for data augmentation for tabular inference . framework generates hypothesis templates transferable to similar tables . authors say framework could generate human-like tabular examples . |
| Outcome: | The proposed framework generates human-like tabular inference examples . it is based on human-written constraints and premise paraphrasing . |
Connecting Attributions and QA Model Behavior on Realistic Counterfactuals (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent research in interpretability of neural models has yielded numerous token attribution techniques, but it is hard to evaluate whether these explanations are faithful. |
| Approach: | They propose to use pairwise attributions to connect outputs to high-level model behavior to examine how well different attribution techniques align with this assumption on realistic counterfactuals in the case of reading comprehension (RC). |
| Outcome: | The proposed methods are better suited to RC than token-level attributions across different RC settings, and the best performance comes from a modification that was proposed to an existing pairwise attribution method. |
Counterfactual reasoning: Testing language models’ understanding of hypothetical scenarios (2023.acl-short)
Copied to clipboard
| Challenge: | Existing language models have demonstrated remarkable improvements in downstream tasks, but it remains unclear to what extent they are consequences of correlation with linguistic heuristics versus robust reasoning about causal relations grounded on understanding of world knowledge. |
| Approach: | They propose to test language models with counterfactual conditionals to test their ability to distinguish hypothetical scenarios from reality. |
| Outcome: | The proposed model overrides real-world knowledge in counterfactual scenarios, but most models are driven by lexical cues. |
Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models (2021.acl-short)
Copied to clipboard
| Challenge: | Pre-trained language models have achieved human-level performance on many Machine Reading Comprehension (MRC) tasks, but it remains unclear whether these models truly understand language or answer questions by exploiting statistical biases in datasets. |
| Approach: | They propose a method to attack MRC models by exposing statistical biases in a RACE dataset and propose an augmented training method that can greatly reduce models’ statistical bias. |
| Outcome: | The proposed method can reduce models’ statistical biases from human-level performance to chance-level. |
A Framework for Evaluation of Machine Reading Comprehension Gold Standards (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing literature on machine reading comprehension (MRC) data is limited on the data design of gold standards. |
| Approach: | They propose a framework to investigate linguistic features, lexical cues and ambiguity in MRC gold standards. |
| Outcome: | The proposed framework investigates the present linguistic features, required reasoning and background knowledge and factual correctness on the one hand, and the presence of lexical cues as a lower bound for the requirement of understanding on the other. |
Probing Neural Network Comprehension of Natural Language Arguments (P19-1)
Copied to clipboard
| Challenge: | Argument Reasoning Comprehension Task (ARCT) focuses on inferences, not just discovering warrants. |
| Approach: | They propose to build an adversarial dataset on which all models achieve random accuracy. |
| Outcome: | The proposed dataset provides a more robust assessment of argument comprehension and should be adopted as the standard in future work. |
REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models have achieved great success on Machine Reading Comprehension (MRC) however, the poor support in evidence extraction hinders them from further advancing MRC. |
| Approach: | They propose a REtrieval-based pre-training approach that strengthens evidence extraction during pre-training by inherited downstream MRC tasks. |
| Outcome: | The proposed approach strengthens evidence extraction during pre-training, which is further inherited by downstream tasks. |
Machine Reading Comprehension as Data Augmentation: A Case Study on Implicit Event Argument Extraction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets are too small to train a model for capturing regularities underlying how event arguments are extracted. |
| Approach: | They propose to bridge implicit EAE with machine reading comprehension (MRC) by building a unified training framework and explicit data augmentation regimes via MRC. |
| Outcome: | The proposed method obtains state-of-the-art performance on two benchmarks and demonstrates superior results in a data-low scenario. |