| Challenge: | In this paper we build a reading comprehension dataset of yes/no questions that are naturally occurring . they often query for complex, non-factoid information, and require difficult entailment-like inference to solve. |
| Approach: | They build a reading comprehension dataset of yes/no questions that are naturally occurring . they find they are unexpectedly challenging and require difficult inferences to solve . |
| Outcome: | The proposed method achieves 80.4% accuracy compared to 90% accuracy of human annotators and 62% majority-baseline. |
Similar Papers
Yes, No or IDK: The Challenge of Unanswerable Yes/No Questions (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing datasets for Yes/No QA are lacking information needed to answer a Yes/Non question. |
| Approach: | They extend the Yes/No QA task by adding questions with an IDK answer to a BoolQ dataset and create out-of-domain test sets for the task. |
| Outcome: | The proposed dataset includes paragraphs together with naturally occurring questions whose answer is either "Yes" or "No". |
Can NLI Models Verify QA Systems’ Predictions? (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent question answering systems perform well on benchmark datasets, but are not always well-calibrated to spot spurious answers under distribution shifts. |
| Approach: | They propose to use natural language inference to verify whether answers are correct . they leverage large pre-trained models and recent prior datasets to construct powerful question conversion and decontextualization modules. |
| Outcome: | The proposed approach improves the confidence estimation of a QA model across different domains, evaluated in a selective QA setting. |
What Makes Reading Comprehension Questions Difficult? (2022.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that natural language understanding benchmarks are not able to measure future progress . a crowdsourcing approach is needed to collect diverse examples without sacrificing diversity or coverage. |
| Approach: | They crowdsource multiple-choice reading comprehension questions for passages from seven sources . they find passage source, length, and readability measures do not significantly affect question difficulty . |
| Outcome: | The results show that passage source, length, and readability measures do not significantly affect question difficulty. |
Language Models Are Poor Learners of Directional Inference (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets fail to test directionality and are infested by artefacts that can be learnt as proxy for entailments, yielding over-optimistic results. |
| Approach: | They propose a benchmark for directional predicate entailments that is extrinsic to existing training sets. |
| Outcome: | The proposed model is incompetent on directional predicate entailments, compared to engorgement graphs, but limited by sparsity. |
BoolQuestions: Does Dense Retrieval Understand Boolean Logic in Language? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Dense retrieval systems focus on optimizing text embedding space while overlooking Boolean logic in language. |
| Approach: | They propose a task to investigate whether retrieval systems can comprehend Boolean logic in language. |
| Outcome: | The proposed method is based on a benchmark dataset covering complex queries containing basic Boolean logic and corresponding annotated passages. |
Benchmarking Query-Conditioned Natural Language Inference (2025.findings-acl)
Copied to clipboard
| Challenge: | Query-conditioned natural language inference (QC-NLI) is a new approach to detect inconsistencies in large language models. |
| Approach: | They propose a task of Query-Conditioned Natural Language Inference to determine the semantic relationship between two documents conditioned on a query. |
| Outcome: | The proposed task is based on a query-conditioned natural language inference (QC-NLI) it is used to determine the relationship between the premise and hypothesis given a given query. |
You Make me Feel like a Natural Question: Training QA Systems on Transformed Trivia Questions (2024.emnlp-main)
Copied to clipboard
Tasnim Kabir, Yoo Yeon Sung, Saptarashmi Bandyopadhyay, Hao Zou, Abhranil Chandra, Jordan Boyd-Graber
| Challenge: | Existing question-answering datasets are expensive and difficult to annotate and time-consuming to gather. |
| Approach: | They propose to transform Manchester questions into web queries using the same question datasets. |
| Outcome: | The proposed questions can be trained on a Manchester QA dataset using the Quiz Bowl (QB) sample. |
Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing (D19-60)
Copied to clipboard
| Challenge: | Workshop on Commonsense Inference in Natural Language Processing focuses on commonsense knowledge representation and application in NLP tasks. |
| Approach: | COIN is a workshop on commonsense inference in natural language processing . workshop included two shared tasks on reading comprehension using commonsensense knowledge . |
| Outcome: | the workshop focused on modeling commonsense knowledge and commonsensing in natural language processing tasks. |
A guide to the dataset explosion in QA, NLI, and commonsense reasoning (2020.coling-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to provide an up-to-date guide to the recent datasets . the target audience is the NLP practitioners who are lost in dozens of the recent data sets. |
| Approach: | This tutorial provides an up-to-date guide to the recent datasets . it surveys old and new methodological issues with dataset construction . |
| Outcome: | This tutorial aims to provide an up-to-date guide to the recent datasets . it surveys the old and new methodological issues with dataset construction . |
How Much Knowledge Can You Pack Into the Parameters of a Language Model? (2020.emnlp-main)
Copied to clipboard
| Challenge: | In this paper, we show that large neural language models trained on unstructured text can attain competitive results on open-domain question answering benchmarks without access to external knowledge. |
| Approach: | They propose to fine-tune pre-trained neural language models to answer questions without external knowledge . they show that this approach scales with model size and performs competitively . |
| Outcome: | The proposed approach scales with model size and performs competitively with open-domain systems that explicitly retrieve answers from an external knowledge source when answering questions. |