WikiHowQA: A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering (2023.acl-long)
Copied to clipboard
| Challenge: | Answering non-factoid questions (NFQs) is a challenging task, requiring passage-level answers that are difficult to construct and evaluate. |
| Approach: | They propose a multi-document NFQA benchmark built on WikiHow, a website dedicated to answering “how-to” questions. |
| Outcome: | The proposed framework includes 11,746 human-written answers along with 74,527 supporting documents. |
Similar Papers
WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts (2025.findings-acl)
Copied to clipboard
Negar Foroutan, Angelika Romanou, Matin Ansaripour, Julian Martin Eisenschlos, Karl Aberer, Rémi Lebret
| Challenge: | Documents are fundamental to preserving and disseminating information, often incorporating complex layouts, tables, and charts that pose significant challenges for automatic document understanding (DU). |
| Approach: | They propose a benchmark for evaluating cross-modal reasoning over tables and charts extracted from 4,000 Wikipedia pages . they evaluate 12 vision-language models that achieve 70% accuracy when provided with direct context . |
| Outcome: | The proposed benchmark evaluates models with high accuracy over tables and charts extracted from 4,000 Wikipedia pages . proprietary models achieve 70% accuracy when provided with direct context, but open-source models perform worse when retrieval from long documents is required. |
RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing QA models are not robust to variations in question constraints, but can be made more robust by tuning on clusters of related questions. |
| Approach: | They introduce RoMQA, the first benchmark for robust, multi-evidence, multianswer question answering (QA) RoMQ contains clusters of related questions that are derived from the Wikidata knowledge graph . |
| Outcome: | The proposed model is the first benchmark for robust, multi-evidence, multianswer question answering (QA) compared to prior QA datasets, it has more human-written questions that require reasoning over more evidence text and have, on average, many more correct answers. |
WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild? (2025.emnlp-main)
Copied to clipboard
An-Lan Wang, Jingqun Tang, Lei Liao, Hao Feng, Qi Liu, Xiang Fei, Jinghui Lu, Han Wang, Hao Liu, Yuliang Liu, Xiang Bai, Can Huang
| Challenge: | Existing benchmarks for document understanding in the wild are based on scanned or digital documents . however, these benchmarks fail to capture the challenges posed by documents in the real world . |
| Approach: | They propose a new benchmark that incorporates a diverse set of manually captured document images reflecting real-world conditions. |
| Outcome: | The proposed model is based on a set of manually captured document images reflecting real-world conditions and is compared with digital or scanned documents. |
ASQA: Factoid Questions Meet Long-Form Answers (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent progress on factoid question answering (QA) does not easily transfer to the task of long-form QA where the goal is to generate detailed explanations. |
| Approach: | They propose a task that focuses on ambiguous factoid questions which have different correct answers depending on interpretation. |
| Outcome: | The proposed metric is reliable and demonstrates agreement between this metric and human judgments, and reveals a considerable gap between human performance and strong baselines. |
FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models (2024.acl-short)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models focus on intradocument dependencies or dependencies between a small number of documents. |
| Approach: | They propose to use a dataset of fan-out question-answer pairs and human-annotated decompositions with English Wikipedia as the knowledge base to evaluate models' reasoning. |
| Outcome: | The proposed dataset shows that models still have room to improve reasoning over inter-document dependencies in a long context. |
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (D18-1)
Copied to clipboard
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, Christopher D. Manning
| Challenge: | Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. |
| Approach: | They propose a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) the questions provide sentence-level supporting facts required for reasoning; and (4) a type of factoid comparison questions to test QA systems’ ability to extract relevant facts and perform necessary comparison. |
| Outcome: | The proposed dataset has 113k Wikipedia-based question-answer pairs and four key features that make it challenging for the latest QA systems. |
CausalQA: A Benchmark for Causal Question Answering (2022.coling-1)
Copied to clipboard
Alexander Bondarenko, Magdalena Wolska, Stefan Heindorf, Lukas Blübaum, Axel-Cyrille Ngonga Ngomo, Benno Stein, Pavel Braslavski, Matthias Hagen, Martin Potthast
| Challenge: | Existing causal question answering datasets are relatively small and only include one type of causal question. |
| Approach: | They construct a benchmark corpus of 1.1 million causal questions with answers . they use a typology derived from a data-driven, manual analysis of QA datasets . |
| Outcome: | The proposed model achieves a ROUGE-L F1 score of 0.48 on the new QA benchmark. |
ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters (N19-1)
Copied to clipboard
| Challenge: | ComQA dataset captures question phenomena and the diverse ways in which they are formulated. |
| Approach: | They propose a large dataset of real user questions that captures question phenomena and the diverse ways in which they are formulated. |
| Outcome: | The proposed dataset can be a driver of future research on factoid question answering (QA). |
IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing open-domain QA tasks focus on questions whose answer can be deduced directly from global factual knowledge. |
| Approach: | They propose a dataset where each question is based on a counterfactual presupposition via an "if" clause. |
| Outcome: | The IfQA dataset contains 3,800 questions that were annotated by crowdworkers on relevant Wikipedia passages. |
LINKAGE: Listwise Ranking among Varied-Quality References for Non-Factoid QA Evaluation via LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Non-factoid (NF) question answering is challenging to evaluate due to diverse potential answers and no objective criterion. |
| Approach: | They propose a listwise NFQA evaluation approach that uses Large Language Models to rank candidate answers in a descending list of reference answers sorted by descending quality. |
| Outcome: | The proposed method has higher correlations with human annotations than standard methods. |