| Challenge: | Recent progress in question answering has been led by neural models, but current methods are too data intensive and weak. |
| Approach: | They propose a multi-choice question answering testbed to encourage research on complex reasoning. |
| Outcome: | The proposed dataset is useful as a benchmark for future work. |
Similar Papers
Towards more equitable question answering systems: How much more data do you need? (2021.acl-short)
Copied to clipboard
| Challenge: | Question answering datasets in English are relatively new, but lack of linguistic diversity in the field is a challenge. |
| Approach: | They propose to use translation and cross-lingual transfer to produce QA systems in multiple languages to improve their performance. |
| Outcome: | The proposed approaches take advantage of existing resources to produce QA systems in multiple languages. |
CRT-QA: A Dataset of Complex Reasoning Question Answering over Tabular Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) show powerful reasoning abilities on text-based tasks, but their reasoning capability on structured data such as tables has not been systematically explored. |
| Approach: | They first establish a comprehensive taxonomy of reasoning and operation types for tabular data analysis and then construct a complex reasoning QA dataset over tabular dataset. |
| Outcome: | The proposed method is able to solve table reasoning tasks without handcrafted demonstrations. |
DBQR-QA: A Question Answering Dataset on a Hybrid of Database Querying and Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Question answering (QA) is a fundamental task in the field of Natural Language Processing (NLP). |
| Approach: | They propose a database querying and reasoning dataset for question answering that is designed to accommodate sequential questions and multi-hop queries. |
| Outcome: | The proposed dataset better mirrors the dynamics of real-world information retrieval and analysis with a particular focus on the financial reports of US companies. |
Discrepancy Detection at the Data Level: Toward Consistent Multilingual Question Answering (2025.emnlp-main)
Copied to clipboard
Lorena Calvo-Bartolomé, Valérie Aldana, Karla Cantarero, Alonso Madroñal de Mesa, Jerónimo Arenas-García, Jordan Lee Boyd-Graber
| Challenge: | Multilingual question answering systems must ensure factual consistency across languages while also accounting for cultural variation in subjective responses. |
| Approach: | They propose a user-in-the-loop fact-checking pipeline to detect factual and cultural discrepancies in multilingual QA knowledge bases. |
| Outcome: | The proposed tool detects factual and cultural discrepancies in bilingual question answering systems. |
Multi-Relational Question Answering from Narratives: Machine Reading and Reasoning in Simulated Worlds (P18-1)
Copied to clipboard
| Challenge: | Question Answering (QA) has primarily focused on knowledge bases or free text as a source of knowledge. |
| Approach: | They propose a task of multi-relational QA over personal narrative using text worlds . they generate and release a lightweight Python-based framework for easily generating additional worlds and narrative . |
| Outcome: | The proposed framework combines elements of structured QA over knowledge bases and unstructured QA . it generates and analyzes five diverse datasets with dynamic narrative . the framework is lightweight and easy to use . |
Bend but Don’t Break? Multi-Challenge Stress Test for QA Models (D19-58)
Copied to clipboard
| Challenge: | a gap remains in reasoning ability compared to a human, and performance tends to degrade when models are exposed to less-constrained tasks. |
| Approach: | They conduct extensive qualitative and quantitative analyses on the results of four models across four datasets . they relate common errors to model capabilities and discuss a way forward . |
| Outcome: | The proposed model performance is based on the results of four models across four datasets. |
EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering (2020.emnlp-main)
Copied to clipboard
| Challenge: | EXAMS is a benchmark dataset for cross-lingual and multilingual question answering for high school examinations. |
| Approach: | They propose to use EXAMS to evaluate cross-lingual and multilingual question answering for high school examinations. |
| Outcome: | The proposed model can be used to explore multilingual reasoning and knowledge transfer methods and pre-trained models in schools in different languages, which was not possible by now. |
MLQA: Evaluating Cross-lingual Extractive Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | Question answering (QA) models have shown rapid progress enabled by the availability of large, high-quality benchmark datasets. |
| Approach: | They present a multi-way aligned extractive QA evaluation benchmark in 7 languages . they evaluate state-of-the-art cross-lingual models and machine-translation-based baselines . |
| Outcome: | The proposed model is based on MLQA, which has over 12K instances in english and 5K in each other language. |
A System for Answering Simple Questions in Multiple Languages (2023.acl-demo)
Copied to clipboard
| Challenge: | Existing knowledge graph question answering systems are limited to simple questions, but they can be used to answer complex questions. |
| Approach: | They propose a multilingual Knowledge Graph Question Answering technique that orders potential responses based on the distance between the question’s text embeddings and the answer’s graph embedds. |
| Outcome: | The proposed method consistently outperforms baseline systems, including seq2seq QA models and complex rule-based pipelines. |
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (D18-1)
Copied to clipboard
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, Christopher D. Manning
| Challenge: | Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. |
| Approach: | They propose a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) the questions provide sentence-level supporting facts required for reasoning; and (4) a type of factoid comparison questions to test QA systems’ ability to extract relevant facts and perform necessary comparison. |
| Outcome: | The proposed dataset has 113k Wikipedia-based question-answer pairs and four key features that make it challenging for the latest QA systems. |