KoRC: Knowledge Oriented Reading Comprehension Benchmark for Deep Text Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for deep text understanding have encountered two major limitations . most require human annotation of knowledge, which leads to limited knowledge coverage . |
| Approach: | They propose a benchmark to help readers understand a document with prior knowledge . they use massive knowledge bases to guide annotators and large language models to construct knowledgable questions . |
| Outcome: | The proposed benchmarks have limited knowledge coverage and use choices or spans as answers, which results in narrow answer space. |
Similar Papers
Beyond Facts- Benchmarking Distributional Reading Comprehension in Large Language Models (2026.findings-acl)
Copied to clipboard
Pei-Fu Guo, Ya An Tsai, Chun-Chia Hsu, Kai-Xin Chen, Yun-Da Tsai, Kai-Wei Chang, Nanyun Peng, Mi-Yen Yeh, Shou-De Lin
| Challenge: | Existing reading comprehension benchmarks focus on factual information, but many real-world tasks require distributional knowledge expressed across text. |
| Approach: | They propose a reading comprehension benchmark for LLMs to evaluate their ability to infer distributional knowledge from natural language. |
| Outcome: | Experiments with multiple LLMs show that the model outperforms baselines, but performance varies widely across distribution types and characteristics. |
DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension (P18-1)
Copied to clipboard
| Challenge: | DuoRC contains 186,089 unique question-answer pairs created from 7680 movie plots . |
| Approach: | They propose a novel dataset for Reading Comprehension that motivates new challenges for neural approaches in language understanding beyond those offered by existing RC datasets. |
| Outcome: | The proposed dataset motivates several new challenges for neural approaches in language understanding beyond those offered by existing RC datasets. |
IIRC: A Dataset of Incomplete Information Reading Comprehension Questions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing reading comprehension tasks focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system’s performance at identifying a potential lack of sufficient information and locating sources for that information. |
| Approach: | They propose to use a dataset with 13K questions over paragraphs from English Wikipedia that provide only partial information to answer them, with the missing information occurring in one or more linked documents. |
| Outcome: | The proposed model achieves 31.1% F1 on the reading comprehension task, while estimated human performance is 88.4%. |
GraCoRe: Benchmarking Graph Comprehension and Complex Reasoning in Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing benchmarks focus primarily on pure graph understanding, lacking a comprehensive evaluation across all graph types and detailed capability definitions. |
| Approach: | They propose a benchmark to evaluate LLMs' graph comprehension and reasoning abilities using a three-tier hierarchical taxonomy and a granular taxonomies. |
| Outcome: | The proposed model includes 11 datasets with 5,140 graphs of varying complexity. |
HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current benchmarks for Hebrew Natural Language Processing (NLP) focus mainly on morpho-syntactic tasks, neglecting the semantic dimension of language understanding. |
| Approach: | They propose to use Hebrew machine reading comprehension (MRC) as extractive Question Answering to address this problem. |
| Outcome: | The proposed benchmark features 30,147 question-answer pairs derived from both Hebrew Wikipedia articles and Israeli tech news. |
Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning (D19-1)
Copied to clipboard
| Challenge: | Existing reading comprehension benchmarks do not contain complex coreferential phenomena . obtaining questions focused on such phenomena is difficult because of lexical cues . |
| Approach: | They propose to use a crowdsourced dataset to examine the ability of models to resolve coreference among entities in Wikipedia paragraphs. |
| Outcome: | The proposed model performs significantly worse than humans on the reading comprehension benchmark . paragraphs and other longer texts typically make multiple references to the same entities . |
BanglaRQA: A Benchmark Dataset for Under-resourced Bangla Language Reading Comprehension-based Question Answering with Diverse Question-Answer Types (2022.findings-emnlp)
Copied to clipboard
Syed Mohammed Sartaj Ekram, Adham Arik Rahman, Md. Sajid Altaf, Mohammed Saidul Islam, Mehrab Mustafy Rahman, Md Mezbaur Rahman, Md Azam Hossain, Abu Raihan Mostofa Kamal
| Challenge: | a lack of diverse and comprehensive question-answering datasets exists in under-resourced languages like Bangla. |
| Approach: | They propose a reading comprehension-based Bangla question-answering dataset . the dataset includes answerable and unanswerable questions covering four categories of questions . |
| Outcome: | The proposed dataset shows that it performs well as a training resource in high-resource languages. |
SciMRC: Multi-perspective Scientific Machine Reading Comprehension (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing datasets focused on single-perspective question-answer pairs overlooking inherent variation in comprehension levels among different readers. |
| Approach: | They propose a multi-perspective scientific machine reading comprehension dataset . their dataset comprises 741 scientific papers and 6,057 question-answer pairs . |
| Outcome: | The proposed dataset includes questions from beginners, students, and experts. |
SKA-Bench: A Fine-Grained Benchmark for Evaluating Structured Knowledge Understanding of LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluations for Structured Knowledge (SK) understanding are non-rigorous and focus on a single type of SK. |
| Approach: | They propose a structured knowledge understanding benchmark that includes four widely used structured knowledge forms. |
| Outcome: | The proposed benchmark is based on four widely used structured knowledge forms . it includes a question, an answer, positive knowledge units, and noisy knowledge units . |
KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoning (2025.acl-long)
Copied to clipboard
Peiqi Sui, Juan Diego Rodriguez, Philippe Laban, J. Dean Murphy, Joseph P. Dexter, Richard Jean So, Samuel Baker, Pramit Chaudhuri
| Challenge: | a study of close reading skills in large language models (LLMs) shows that LLMs still lag behind human evaluators on 10 of 11 tasks. |
| Approach: | They propose a benchmark to evaluate close reading skills in large language models . they propose three tasks to approximate different elements of the close reading process . |
| Outcome: | The proposed benchmarks show that state-of-the-art LLMs possess some college-level close reading competency, but performance still trails human evaluators on 10 out of 11 tasks. |