SCITAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Types (2025.findings-acl)
Copied to clipboard
Xuanliang Zhang, Dingzirui Wang, Baoxin Wang, Longxu Dou, Xinyuan Lu, Keyan Xu, Dayong Wu, Qingfu Zhu
| Challenge: | Existing scientific question answering datasets lack diverse reasoning types and neglect relevance between tables and text. |
| Approach: | They propose a scientific question answering benchmark for scientific tables and text with diverse reasoning types (SCITAT) to address these challenges, they propose QA benchmark which incorporates tables and texts to ensure that the questions encompass both tables and textes. |
| Outcome: | The proposed benchmark improves by 4.1% over baselines on SCITAT. |
Similar Papers
How Robust Are the QA Models for Hybrid Scientific Tabular Data? A Study Using Customized Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing tabular QA models are lacking in understanding their robustness on scientific information. |
| Approach: | They propose a dataset to assess the robustness of tabular QA models on scientific hybrid tabular data. |
| Outcome: | The proposed model performs well on scientific tables and text, while the best score is 0.462. |
Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Table Question Answering (TQA) aims to answer natural language questions using tabular data. |
| Approach: | They propose a systematic overview of TQA research using large language models and summarize available benchmarks based on task features. |
| Outcome: | The proposed framework provides a comprehensive overview of the current state of the art in the field of Table Question Answering. |
AIT-QA: Question Answering Dataset over Complex Tables in the Airline Industry (2022.naacl-industry)
Copied to clipboard
Yannis Katsis, Saneem Chemmengath, Vishwajeet Kumar, Samarth Bharadwaj, Mustafa Canim, Michael Glass, Alfio Gliozzo, Feifei Pan, Jaydeep Sen, Karthik Sankaranarayanan, Soumen Chakrabarti
| Challenge: | Table Question Answering (Table QA) systems have been shown to be highly accurate when trained and tested on open-domain datasets built on top of Wikipedia tables. |
| Approach: | They propose a domain-specific Table QA test dataset to test Table Question Answering systems on open-domain datasets built on top of Wikipedia tables. |
| Outcome: | The proposed methods are highly accurate when tested on open-domain datasets built on top of Wikipedia tables. |
GRI-QA: a Comprehensive Benchmark for Table Question Answering over Environmental Data (2025.findings-acl)
Copied to clipboard
Michele Luca Contalbo, Sara Pederzoli, Francesco Del Buono, Venturelli Valeria, Francesco Guerra, Matteo Paganelli
| Challenge: | Existing Table Question Answering systems for environmental reporting are complex and require specialized terminology and complex questions. |
| Approach: | They propose a test benchmark to evaluate Table QA approaches in the environmental domain. |
| Outcome: | The proposed benchmark reveals a significant gap between human and model performance in multi-step reasoning. |
A guide to the dataset explosion in QA, NLI, and commonsense reasoning (2020.coling-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to provide an up-to-date guide to the recent datasets . the target audience is the NLP practitioners who are lost in dozens of the recent data sets. |
| Approach: | This tutorial provides an up-to-date guide to the recent datasets . it surveys old and new methodological issues with dataset construction . |
| Outcome: | This tutorial aims to provide an up-to-date guide to the recent datasets . it surveys the old and new methodological issues with dataset construction . |
FREB-TQA: A Fine-Grained Robustness Evaluation Benchmark for Table Question Answering (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for Table Question Answering lack robustness . current systems generate inconsistent responses by performing coarse-grained changes in tables and questions. |
| Approach: | They formalize three major desiderata for a fine-grained evaluation of robustness of TQA systems . they aim to evaluate TQA responses on the content of relevant cells rather than on biases . |
| Outcome: | The proposed benchmarks show that state-of-the-art TQA systems fail in three aspects . the benchmarks are released publicly and are expected to improve over time . |
SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers (2024.emnlp-main)
Copied to clipboard
| Challenge: | SciDQA is a dataset for question-answering that challenges language models to deeply understand scientific articles. |
| Approach: | They propose a new dataset for reading comprehension that challenges language models to deeply understand scientific articles consisting of 2,937 QA pairs. |
| Outcome: | The SciDQA dataset is based on 2,937 QA pairs and decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering. |
LILA: A Unified Benchmark for Mathematical Reasoning (2022.emnlp-main)
Copied to clipboard
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, Ashwin Kalyan
| Challenge: | Towards evaluating and improving AI systems in this domain, we propose a mathematical reasoning benchmark based on 23 diversetasks . |
| Approach: | They propose a mathematical reasoning benchmark that includes 23 diverse tasks . they extend the benchmark by collecting task instructions and solutions in the form of Python programs . |
| Outcome: | The proposed model improves on multi-tasking while the best performing model only achieves 60.40%. |
SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for multimodal large language models fail to capture complexity and traceability of reasoning processes . SciVQR includes domain-specific visuals and challenges models to combine visual comprehension with reasoning. |
| Approach: | They propose a multimodal benchmark for scientific reasoning covering 54 subfields . SciVQR includes domain-specific visuals and challenges models to combine visual comprehension with reasoning . |
| Outcome: | SciVQR evaluates 54 subfields in mathematics, physics, chemistry, geography, astronomy, and biology . the results highlight the need for improved multi-step reasoning and integration of interdisciplinary knowledge . |
MULTITAT: Benchmarking Multilingual Table-and-Text Question Answering (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing TATQA datasets are limited to English, leading to drawbacks . existing datasets overlook challenges of multilingual TAT-QA and do not reflect real-world multilingual scenarios . |
| Approach: | They propose a multilingual TATQA dataset that can be translated into 10 languages . they use data from 3 mainstream TATQ datasets and analyze the results . |
| Outcome: | The proposed dataset outperforms other baselines by an average of 3.3 . |