ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing question answering datasets for common sense reasoning are lacking for prototypical situations. |
| Approach: | They propose a question answering dataset for training and evaluating common sense reasoning capabilities of artificial intelligence systems in such prototypical situations. |
| Outcome: | The proposed model outperforms existing models on all evaluation metrics with a meaningful gap. |
Similar Papers
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (N19-1)
Copied to clipboard
| Challenge: | Recent work on question answering relies on factoid questions with little general knowledge. |
| Approach: | They propose a dataset to capture commonsense question answering with prior knowledge . they extract multiple-choice questions that discriminate between the source and target concepts . |
| Outcome: | The proposed dataset captures commonsense reasoning beyond associations . it obtains 56% accuracy, well below human performance, which is 89% . |
CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to answer open domain questions rely on unlabeled text or synthetically generated question-answer pairs. |
| Approach: | They propose a large-scale open-domain question-answering dataset based on the Common Crawl project that can be used to in-domain pre-train popular language models. |
| Outcome: | The proposed dataset achieves promising results in zero-shot, low resource and fine-tuned settings across multiple tasks, models and benchmarks. |
ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters (N19-1)
Copied to clipboard
| Challenge: | ComQA dataset captures question phenomena and the diverse ways in which they are formulated. |
| Approach: | They propose a large dataset of real user questions that captures question phenomena and the diverse ways in which they are formulated. |
| Outcome: | The proposed dataset can be a driver of future research on factoid question answering (QA). |
ProtoVQA: An Adaptable Prototypical Framework for Explainable Fine-Grained Visual Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Question Answering (VQA) is increasingly used in diverse applications where models must provide accurate answers and explanations that humans can easily understand and verify. |
| Approach: | They propose a unified prototypical framework that learns question-aware prototypes that serve as reasoning anchors and applies spatially constrained matching to ensure that the selected evidence is coherent and semantically relevant. |
| Outcome: | The proposed framework yields faithful, fine-grained explanations while maintaining competitive accuracy. |
QuAC: Question Answering in Context (D18-1)
Copied to clipboard
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, Luke Zettlemoyer
| Challenge: | a dataset for Question Answering in Context contains 14K information-seeking QA dialogs . questions are often more open-ended, unanswerable, or only meaningful within the dialog context . |
| Approach: | They propose a dataset for Question Answering in Context that contains 14K dialogs . they use a student to ask questions about a Wikipedia section and a teacher to answer them . |
| Outcome: | The proposed dataset underperforms humans in a number of reference models . the dataset contains 14K information-seeking dialogs over sections from Wikipedia . |
Few-Shot Complex Knowledge Base Question Answering via Meta Reinforcement Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to complex question-answering (CQA) exhibit uneven performance when questions have different types, harboring inherently different characteristics, e.g., difficulty level. |
| Approach: | They propose a meta-reinforcement learning approach to program induction in CQA to tackle the potential distributional bias in questions. |
| Outcome: | The proposed method achieves state-of-the-art performance on the CQA dataset while using only five trial trajectories for the top-5 retrieved questions in each support set. |
HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing question answering datasets focus on dealing with homogeneous information, but using homogenous information alone might lead to coverage problems. |
| Approach: | They propose a large-scale question-answering dataset that requires reasoning on heterogeneous information. |
| Outcome: | The proposed model can achieve an EM score of 40% while the existing model is far behind human performance. |
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (D18-1)
Copied to clipboard
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, Christopher D. Manning
| Challenge: | Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. |
| Approach: | They propose a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) the questions provide sentence-level supporting facts required for reasoning; and (4) a type of factoid comparison questions to test QA systems’ ability to extract relevant facts and perform necessary comparison. |
| Outcome: | The proposed dataset has 113k Wikipedia-based question-answer pairs and four key features that make it challenging for the latest QA systems. |
PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations (2023.findings-acl)
Copied to clipboard
| Challenge: | Mars? - PragmatiCQA |
| Approach: | Mars? - The Paper . |
| Outcome: | The proposed dataset features 6873 QA pairs that explores pragmatic reasoning in conversations over a diverse set of topics. |
An empirical analysis of existing systems and datasets toward general simple question answering (2020.coling-main)
Copied to clipboard
| Challenge: | evaluators of simple factoid question answering using different datasets are not able to solve SimpleQuestions. |
| Approach: | They evaluate the progress of the field toward solving simple factoid questions over a knowledge base. |
| Outcome: | The proposed model is nearly solved on the most popular dataset, but not on the robustness of existing systems. |