Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj, Rajarshi Das, Andrew McCallum, Maria Chang, Achille Fokoue, Pavan Kapanipathi, Nicholas Mattei, Ryan Musa, Kartik Talamadupula, Michael Witbrock
| Challenge: | a new interface for human annotation of science question-answer pairs with their knowledge and reasoning types is proposed . the interface is based on previous work on the ARC dataset, but does not provide clear definitions of these types of knowledge. |
| Approach: | They propose an interface for human annotation of science question-answer pairs with their respective knowledge and reasoning types. |
| Outcome: | The proposed interface improves the classification of science questions in a preliminary study involving 10 participants. |
Similar Papers
Improving Question Answering with External Knowledge (D19-58)
Copied to clipboard
| Challenge: | ARC-Easy, ARC Challenge, and OpenBookQA use Wikipedia to augment training data . performance degrades when additional instances exhibit higher difficulty than original training data. |
| Approach: | They propose two methods for exploiting external knowledge for QA in science . they enrich the original corpus with relevant text snippets from an open-domain resource . the second method simply increases the amount of training data by appending additional in-domain instances. |
| Outcome: | The proposed methods achieve gains in accuracy of 8.1%, 13.0%, and 12.8% on science QA tasks. |
QUADRo: Dataset and Models for QUestion-Answer Database Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing tools do not consider answers (question-question similarity only) or their quality in the annotation process. |
| Approach: | They propose an open-domain annotated resource that retrieves questions from pre-computed databases . they use a binary label associated with each question/answer pair to determine relevance . |
| Outcome: | The proposed model trains and evaluates models on 443,000 examples with respect to key aspects of QA systems. |
CRT-QA: A Dataset of Complex Reasoning Question Answering over Tabular Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) show powerful reasoning abilities on text-based tasks, but their reasoning capability on structured data such as tables has not been systematically explored. |
| Approach: | They first establish a comprehensive taxonomy of reasoning and operation types for tabular data analysis and then construct a complex reasoning QA dataset over tabular dataset. |
| Outcome: | The proposed method is able to solve table reasoning tasks without handcrafted demonstrations. |
A guide to the dataset explosion in QA, NLI, and commonsense reasoning (2020.coling-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to provide an up-to-date guide to the recent datasets . the target audience is the NLP practitioners who are lost in dozens of the recent data sets. |
| Approach: | This tutorial provides an up-to-date guide to the recent datasets . it surveys old and new methodological issues with dataset construction . |
| Outcome: | This tutorial aims to provide an up-to-date guide to the recent datasets . it surveys the old and new methodological issues with dataset construction . |
Automatic Annotation of Semantic Term Types in the Complete ACL Anthology Reference Corpus (L18-1)
Copied to clipboard
| Challenge: | a recent increase in quantitative studies of scientific text collections has led to a significant increase in the use of semantic labeling techniques. |
| Approach: | They propose to use semantic class labels to enhance a well-known resource . they use semantic labels to assign semantic class labeling to technical terms . |
| Outcome: | The proposed approach enhances the ACL Anthology Reference Corpus with semantic class labels for 20,000 technical terms . the goal is to use this information as one feature in the profiling of scientific papers, communities, and disciplines. |
ScheMatiQ: From Research Question to Structured Data through Interactive Schema Discovery (2026.acl-demo)
Copied to clipboard
| Challenge: | a new approach to natural-language research questions requires manual effort to generate an annotation schema and label the corpus. |
| Approach: | They propose a natural-language search tool that takes a question and a corpus to produce a schema and db with a web interface that lets steer and revise the extraction. |
| Outcome: | The proposed model yields outputs that support real-world analysis in law and computational biology. |
WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-hop Inference (L18-1)
Copied to clipboard
| Challenge: | Existing methods of automated inference do not provide enough gold explanations to train models . standardized science exams are a challenge task for question answering . |
| Approach: | They propose to manually construct a corpus of explanations for standardized science exams . they also provide an explanation-centered tablestore that contains the knowledge to construct these explanations . |
| Outcome: | The proposed model provides detailed explanations for standardized science exams . the authors show that the proposed model can be trained on the basis of gold explanations . |
Ai2 Scholar QA: Organized Literature Synthesis with Attribution (2025.acl-demo)
Copied to clipboard
Amanpreet Singh, Joseph Chee Chang, Dany Haddad, Aakanksha Naik, Jena D. Hwang, Rodney Kinney, Daniel S Weld, Doug Downey, Sergey Feldman
| Challenge: | Ai2 Scholar QA is a free online scientific question answering application . it uses retrieval-augmented generation to answer complex scientific questions . many of these systems are expensive to use and closed-source . |
| Approach: | They propose a retrieval-augmented generation-based scientific question answering application . it uses a Python package and an interactive web app to make the entire pipeline public . they compare it with other similar question-answering applications . |
| Outcome: | The proposed system outperforms other systems on a recent scientific QA benchmark. |
Towards Multi-Document Question Answering in Scientific Literature: Pipeline, Dataset, and Evaluation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing QA systems do not strictly enforce cross-document synthesis or exploit the explicit inter-paper structure that links sources. |
| Approach: | They propose a pipeline methodology for constructing a multi-document academic QA dataset . they detect communities based on citation networks and leverage Large Language Models . |
| Outcome: | The proposed method generates QA pairs related to multi-document content automatically and forms coherent communities based on citation networks and large language models. |
Interactive Classification by Asking Informative Questions (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for intent classification rely on a single user input and do not interact with the user to reduce ambiguity and improve the final prediction. |
| Approach: | They propose a limited form of interaction to natural language intent classification . they add binary or multi-choice questions to the system to ask missing information . |
| Outcome: | The proposed method can be bootstrapped without interaction data and is scalable to two domains. |