Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (D18-1)
Copied to clipboard
| Challenge: | Existing QA datasets focus on linguistic understanding, but OpenBookQA probes deeper understanding of topic and language. |
| Approach: | They propose a dataset modeled after open book exams for question answering . the open book is a set of 1326 elementary level science facts . human performance on OpenBookQA is close to 92%, they show . |
| Outcome: | The proposed dataset is modeled after open book exams for question answering . human performance on OpenBookQA is close to 92%, but many state-of-the-art QA methods perform poorly . |
Similar Papers
Careful Selection of Knowledge to Solve Open Book Question Answering (P19-1)
Copied to clipboard
| Challenge: | Open book question answering requires deeper reasoning involving linguistic understanding and common knowledge. |
| Approach: | They propose a dataset that mimics open book question answering to achieve 72.0% accuracy. |
| Outcome: | The proposed dataset achieves 72.0% accuracy, an 11.6% improvement over the current state of the art. |
Book QA: Stories of Challenges and Opportunities (D19-58)
Copied to clipboard
| Challenge: | Existing approaches to answer questions based on the full text of books are limited by their unique characteristics. |
| Approach: | They propose a system for answering questions based on the full text of books . they use a memory network to reason and predict an answer, and a novel question generator to improve generalization. |
| Outcome: | The proposed system improves on the recently published NarrativeQA corpus on Who questions . it shows that the proposed system is highly challenging and needs more research . |
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (D18-1)
Copied to clipboard
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, Christopher D. Manning
| Challenge: | Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. |
| Approach: | They propose a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) the questions provide sentence-level supporting facts required for reasoning; and (4) a type of factoid comparison questions to test QA systems’ ability to extract relevant facts and perform necessary comparison. |
| Outcome: | The proposed dataset has 113k Wikipedia-based question-answer pairs and four key features that make it challenging for the latest QA systems. |
SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers (2024.emnlp-main)
Copied to clipboard
| Challenge: | SciDQA is a dataset for question-answering that challenges language models to deeply understand scientific articles. |
| Approach: | They propose a new dataset for reading comprehension that challenges language models to deeply understand scientific articles consisting of 2,937 QA pairs. |
| Outcome: | The SciDQA dataset is based on 2,937 QA pairs and decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering. |
Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent advances in open-domain question answering (ODQA) have led to human-level performance on many datasets. |
| Approach: | They provide a comprehensive and quantitative analysis about the difficulty of book QA . they compare the results of their research with extensive ODQA experiments . |
| Outcome: | The proposed model outperforms existing models on event-oriented questions on the NarrativeQA dataset. |
A guide to the dataset explosion in QA, NLI, and commonsense reasoning (2020.coling-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to provide an up-to-date guide to the recent datasets . the target audience is the NLP practitioners who are lost in dozens of the recent data sets. |
| Approach: | This tutorial provides an up-to-date guide to the recent datasets . it surveys old and new methodological issues with dataset construction . |
| Outcome: | This tutorial aims to provide an up-to-date guide to the recent datasets . it surveys the old and new methodological issues with dataset construction . |
ELQA: A Corpus of Metalinguistic Questions and Answers about English (2023.acl-long)
Copied to clipboard
| Challenge: | ELQA corpus is metalinguistic—it consists of language about language. |
| Approach: | They present a corpus of questions and answers in and about the English language . they use a free-form question answering task and multiple LLMs to analyze their capacity . |
| Outcome: | The ELQA corpus covers grammar, meaning, fluency, and etymology . the results can be used to investigate metalinguistic capabilities of NLU models . |
PolQA: Polish Question Answering Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent proposed systems for open-domain question answering (OpenQA) require large amounts of training data to achieve state-of-the-art performance. |
| Approach: | They propose an efficient annotation strategy that increases passage retrieval accuracy@10 by 10.55 p.p. while reducing the annotation cost by 82%. |
| Outcome: | The proposed approach increases passage retrieval accuracy @10 by 10.55 p.p. while reducing the annotation cost by 82%. |
Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA? (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work is limited in using small benchmarks with high test-train overlaps. |
| Approach: | They construct a dataset of closed-book QA using SQuAD and investigate the performance of BART. |
| Outcome: | Experiments show that pre-trained language models can achieve high performance on closed-book QA tasks. |
Open-WikiTable : Dataset for Open Domain Question Answering with Complex Reasoning over Table (2023.findings-acl)
Copied to clipboard
| Challenge: | Open-WikiTable is the first open domain question answering dataset that requires complex reasoning over tables. |
| Approach: | They propose to use open-domain question answering over tables to extract questions from tables. |
| Outcome: | The dataset is publicly available. it is built upon WikiSQL and WikiTableQuestions. |