Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent efforts to create challenge benchmarks that test the abilities of natural language understanding models have largely depended on human annotations. |
| Approach: | They propose a framework for automatic reasoning-oriented perturbation of question-answer pairs that decomposes a question into reasoning steps required to answer it and generates new question-anchor pairs. |
| Outcome: | The proposed framework generates evaluation sets for reading comprehension benchmarks and generates examples without human intervention. |
Similar Papers
Break It Down: A Question Understanding Benchmark (2020.tacl-1)
Copied to clipboard
| Challenge: | Understanding natural language questions entails the ability to break down a question into the requisite steps for computing its answer. |
| Approach: | They introduce a Question Decomposition Meaning Representation (QDMR) for questions . they demonstrate that QDMRs can be annotated at scale using a hotpotQA dataset . |
| Outcome: | The proposed model outperforms several natural baselines in the open-domain question answering hotpotQA dataset and can be deterministically converted to a pseudo-SQL formal language. |
Explaining Answers with Entailment Trees (2021.emnlp-main)
Copied to clipboard
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, Peter Clark
| Challenge: | ENTAILMENTBANK is the first dataset to contain multistep entailment trees. |
| Approach: | They propose to generate explanations in the form of entailment trees, a tree of multipremise entanglements steps from facts that are known to the hypothesis of interest. |
| Outcome: | The proposed model can generate explanations in the form of entailment trees . this is a tree of multipremise enttailment steps from facts known to the hypothesis of interest. |
Contextualized Word Representations for Reading Comprehension (N18-2)
Copied to clipboard
| Challenge: | Reading comprehension (RC) is a high-level task in natural language understanding that requires reading a document and answering questions about its content. |
| Approach: | They propose to provide a standard neural network for reading a document and answering a question about its content. |
| Outcome: | The proposed model improves on the competitive SQuAD dataset by providing rich contextualized word representations and allowing it to choose between context-dependent and context-independent representations. |
TabPert : An Effective Platform for Tabular Perturbation (2021.emnlp-demo)
Copied to clipboard
| Challenge: | Current transformers-based models outperform humans on factual evidence evaluation tasks when presented as simple unstructured text. |
| Approach: | TabPert generates counterfactual data to assess model tabular reasoning issues. |
| Outcome: | TabPert analyzes the model's shortcomings methodically and quantitatively. |
Summarize-then-Answer: Generating Concise Explanations for Multi-hop Reading Comprehension (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to multi-hop Reading Comprehension (RC) have greatly improved its explainability, models ability to explain their own answers. |
| Approach: | They propose to generate a question-focused abstractive summary of input paragraphs and feed it to an RC system. |
| Outcome: | The proposed explanation generates more compact explanations than an extractive explainer with limited supervision while maintaining sufficiency. |
Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answering (2020.emnlp-main)
Copied to clipboard
| Challenge: | despite rapid progress in multihop question-answering, models still have trouble explaining why an answer is correct. |
| Approach: | They propose three explanation datasets in which explanations from corpus facts are annotated . they first annotate multiple candidate explanations for each answer, then use crowd-sourcing perturbations to test generalization . |
| Outcome: | The proposed datasets improve explanation quality but still behind the upper bound . the proposed dataset can be used to improve explanations using a BERT-based classifier . |
Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to end-to-end questionanswering assume that pre-trained language can decompose complex tasks into more straightforward sub-tasks. |
| Approach: | They propose to use distant supervision to train decomposition-based transformers for large-scale parallel news. |
| Outcome: | The proposed model improves on semantic parsing and on hotpotQA and strategyQA datasets by 20% to 30%. |
CoReTab: Improving Multimodal Table Understanding with Code-driven Reasoning (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing datasets for multimodal table understanding provide short factual answers without explicit multi-step reasoning supervision. |
| Approach: | They propose a code-driven reasoning framework that produces scalable, interpretable, and automatically verifiable annotations by coupling multi-step reasoning with executable Python code. |
| Outcome: | The proposed model achieves significant gains over baseline models while producing transparent and verifiable reasoning traces. |
Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts (2022.emnlp-main)
Copied to clipboard
| Challenge: | Question-answering datasets require a broad set of reasoning skills. |
| Approach: | They use QDMR representations to programmatically create hard-to-cheat synthetic contexts for real questions in multi-step reasoning datasets. |
| Outcome: | The proposed model improves performance by 13 F1 points on 4 multi-step QA datasets and 21 points on more complex questions. |
Generative Interpretation: Toward Human-Like Evaluation for Educational Question-Answer Pair Generation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing evaluation methods often fail to produce objective results and favor high similarity to the ground-truth question-answer pairs. |
| Approach: | They propose an alternative approach to evaluate question-answer generation using Generative Interpretation (GI) GI outperforms existing evaluation methods in terms of human alignment . |
| Outcome: | The proposed approach outperforms existing evaluation methods in human alignment and shows comparable performance with GPT3.5, only with BART-large. |