FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models (2024.acl-short)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models focus on intradocument dependencies or dependencies between a small number of documents. |
| Approach: | They propose to use a dataset of fan-out question-answer pairs and human-annotated decompositions with English Wikipedia as the knowledge base to evaluate models' reasoning. |
| Outcome: | The proposed dataset shows that models still have room to improve reasoning over inter-document dependencies in a long context. |
Similar Papers
JEMHopQA: Dataset for Japanese Explainable Multi-Hop Question Answering (2024.lrec-main)
Copied to clipboard
| Challenge: | a dataset for explainable QA in Japanese is available for many languages, but not in other languages. |
| Approach: | They present a multi-hop QA dataset based on Japanese Wikipedia . it includes question-answer pairs and supporting evidence in the form of derivation triples . they show that the dataset is sufficiently challenging for state-of-the-art LLMs based upon this dataset . |
| Outcome: | The proposed dataset is based on Japanese Wikipedia and can be used to evaluate QA tasks. |
WikiHowQA: A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering (2023.acl-long)
Copied to clipboard
| Challenge: | Answering non-factoid questions (NFQs) is a challenging task, requiring passage-level answers that are difficult to construct and evaluate. |
| Approach: | They propose a multi-document NFQA benchmark built on WikiHow, a website dedicated to answering “how-to” questions. |
| Outcome: | The proposed framework includes 11,746 human-written answers along with 74,527 supporting documents. |
GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing LLM leaderboards often reference scores reported in other papers without consistent settings and prompts, which may encourage cherry-picking favored settings and for better results. |
| Approach: | They propose an open-source and reproducible LLM evaluation suite built on top of OpenAI Evals that systematically evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
| Outcome: | The evaluation suite is built on top of OpenAI Evals and evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
MEBench: Benchmarking Large Language Models for Cross-Document Multi-Entity Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) and Retrieval-augmented Generation (RAG) systems show promise, but their performance on cross-document MEQA remains underexplored due to the lack of tailored benchmarks. |
| Approach: | They propose a scalable multi-document, multi-entity benchmark to evaluate LLMs' capacity to retrieve, consolidate, and reason over scattered and dense information. |
| Outcome: | The proposed benchmarks show that even advanced models achieve only 59% accuracy on MEBench. |
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (D18-1)
Copied to clipboard
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, Christopher D. Manning
| Challenge: | Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. |
| Approach: | They propose a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) the questions provide sentence-level supporting facts required for reasoning; and (4) a type of factoid comparison questions to test QA systems’ ability to extract relevant facts and perform necessary comparison. |
| Outcome: | The proposed dataset has 113k Wikipedia-based question-answer pairs and four key features that make it challenging for the latest QA systems. |
DebateQA: Evaluating Question Answering on Debatable Knowledge (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing QA benchmarks that provide fixed answers to debatable questions are inadequate for evaluating their performance. |
| Approach: | They propose to use a dataset of 2,941 debatable questions to assess their ability to provide comprehensive answers to inherently debatably asked questions. |
| Outcome: | The proposed model performs well on 2,941 debatable questions accompanied by human-annotated partial answers that capture a variety of perspectives. |
ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use (2025.acl-long)
Copied to clipboard
Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu, Zhiheng Xi, Siyu Yuan, Tao Gui, Qi Zhang, Xuanjing Huang, Jiecao Chen
| Challenge: | Effective evaluation of multi-hop tool use is critical for analyzing the understanding, reasoning, and function-calling capabilities of large language models. |
| Approach: | They propose a dataset that provides rigorous evaluation of multi-hop tool use. |
| Outcome: | The proposed model achieves 49.04% accuracy across five model families. |
M³GQA: A Multi-Entity Multi-Hop Multi-Setting Graph Question Answering Benchmark (2025.acl-long)
Copied to clipboard
| Challenge: | GraphRAG systems have achieved remarkable progress in enhancing performance and reliability of large language models. |
| Approach: | They propose a GraphRAG benchmark focusing on multi-entity queries with six settings for comprehensive evaluation. |
| Outcome: | The proposed method can construct diverse data with semantically correct ground-truth reasoning paths. |
How Accurate Are LLMs at Multi-Question Answering on Conversational Transcripts? (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are used for question answering over long contexts . high computational costs and latency hinder the process . |
| Approach: | They explore the capabilities of Large Language Models to answer multiple questions based on the same conversational context. |
| Outcome: | The proposed models outperform proprietary and public models in question answering . their results show that they can be cost-effective and transparent . |
TableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing TableQA benchmarks focus on simple flat tables and suffer from data leakage . current benchmarks are monolingual and fail to capture cross-lingual variability . |
| Approach: | They propose a table-based TableQA benchmark to evaluate LLMs on real-world tasks. |
| Outcome: | The proposed benchmarks show that they achieve high agreement with human judgment . the proposed framework improves on the alignment between model responses and reference answers . |