Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA (P19-1)
Copied to clipboard
| Challenge: | In multi-hop question answering, models need to connect multiple pieces of evidence scattered in a long context to answer the question. |
| Approach: | They propose to use a control unit that dynamically attends to the question at different reasoning hops to guide the model's multi-hop reasoning. |
| Outcome: | The proposed model outperforms baseline models but is limited on adversarial test. |
Similar Papers
Robustifying Multi-hop QA through Pseudo-Evidentiality Training (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to robustify multi-hop question answering models require expensive annotations. |
| Approach: | They propose a method to supervise answers with right reasoning chains without annotations . they compare answers confidence with and without evidence sentences to generate "pseudo-evidentiality" annotations. |
| Outcome: | The proposed model is accurate and robust in multi-hop reasoning. |
Generative Context Pair Selection for Multi-hop Question Answering (2021.emnlp-main)
Copied to clipboard
Dheeru Dua, Cicero Nogueira dos Santos, Patrick Ng, Ben Athiwaratkun, Bing Xiang, Matt Gardner, Sameer Singh
| Challenge: | Recent studies have shown that discriminative training results in models that exploit these underlying biases to achieve a better held-out performance, without learning the right way to reason. |
| Approach: | They propose a generative context selection model for multi-hop QA that reasons about how the given question could have been generated given a context pair and not just independent contexts. |
| Outcome: | The proposed model outperforms the state-of-the-art model on hotpotQA while being comparable to the state of the art answering performance on adversarial held-out set. |
Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies have utilized underlying reasoning (UR) tasks in multi-hop question answering datasets to explain the predicted answers and evaluate models' reasoning abilities. |
| Approach: | They analyze UR tasks in QA datasets to determine their effectiveness . they find that UR task is helpful in preventing reasoning shortcuts . |
| Outcome: | The proposed model improves QA performance, reasoning shortcuts, and robustness on adversarial questions. |
Understanding Dataset Design Choices for Multi-hop Reasoning (N19-1)
Copied to clipboard
| Challenge: | Existing datasets that explicitly focus on multi-hop reasoning are lacking in learning multi-tasking. |
| Approach: | They propose to use sentence-factored models to solve multi-hop question answering tasks . they find spurious correlations in unmasked versions of WikiHop and HotpotQA . |
| Outcome: | The proposed datasets are used to test models on multi-hop question answering tasks. |
Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts? (2025.findings-acl)
Copied to clipboard
| Challenge: | Latent multi-hop reasoning is a problem in Large Language Models that can develop shortcuts by encountering the head entity and answer entity in training sequences. |
| Approach: | They propose desiderata for shortcut-free evaluation of latent multi-hop reasoning ability . they exclude test queries where head and answer entities might have co-appeared . |
| Outcome: | The proposed model can latently recall and compose single-hop facts without shortcuts, but only for certain types of queries. |
Understanding and Improving Zero-shot Multi-hop Reasoning in Generative Question Answering (2022.coling-1)
Copied to clipboard
| Challenge: | Generative question answering (QA) models generate answers to complex questions, but their mechanism for doing so is still poorly understood. |
| Approach: | They decompose multi-hop questions into multiple corresponding single-hop question chains and find marked inconsistency in QA models’ answers on these pairs of ostensibly identical question chains. |
| Outcome: | The proposed models lack zero-shot multi-hop reasoning ability when trained on single-hop questions and on logical forms. |
Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models exploit dataset artifacts to produce correct answers without connecting information across multiple facts. |
| Approach: | They formalize disconnected reasoning across subsets of supporting facts to reduce disconnected reasoning . they propose an automatic transformation of existing datasets that reduces disconnected reasoning. |
| Outcome: | The proposed model-agnostic probe reduces disconnected reasoning in a reading comprehension setting. |
Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps (2020.coling-main)
Copied to clipboard
| Challenge: | Existing multi-hop question answering datasets do not provide a complete explanation for the reasoning process from the question to the answer. |
| Approach: | They propose a multi-hop question answering dataset that uses structured and unstructured data to test reasoning skills. |
| Outcome: | The proposed dataset ensures multi-hop reasoning while being challenging for multi-models. |
Low-Resource Generation of Multi-hop Reasoning Questions (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods to generate valid and fluent questions from text are limited and insufficient for training. |
| Approach: | They propose to generate multi-hop reasoning questions from the raw text in a low resource circumstance by deducing over multiple relations on several sentences in the text. |
| Outcome: | The proposed model can be applied to the task of machine reading comprehension and achieve significant performance improvements. |
Can We Guide a Multi-Hop Reasoning Language Model to Incrementally Learn at Each Single-Hop? (2022.coling-1)
Copied to clipboard
| Challenge: | Recent developments have shown that pre-trained language models are effective soft reasoners over language. |
| Approach: | They propose to model multi-hop reasoning process as a sequence of explicit single-hop steps. |
| Outcome: | The proposed model improves on multiple-choice question answering and reading comprehension with 68.4% and 16.0% w.r.t. classic PLMs. |