| Challenge: | Existing cloze tests are used to evaluate language proficiency in language exams, but they are not yet available. |
| Approach: | They propose to create a large-scale human-created cloze test dataset CLOTH, which contains questions used in middle-school and high-school language exams. |
| Outcome: | The proposed dataset outperforms existing models and shows that humans outperformed existing models by a significant margin. |
Similar Papers
ClozEx: A Task toward Generation of English Cloze Explanation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing tasks and datasets specifically designed for generating language learner explanations for cloze questions are lacking . clozing questions are used to assess language proficiency and enhance language learning . |
| Approach: | They propose a task ClozEx to generate explanations for cloze questions in LA . they use a curated dataset of clozing questions paired with explanations . |
| Outcome: | The proposed task generates fluent explanations for cloze questions in English as a second language learners. |
ChID: A Large-scale Chinese IDiom Dataset for Cloze Test (P19-1)
Copied to clipboard
| Challenge: | cloze-style reading comprehension in Chinese is limited due to the lack of various corpora. |
| Approach: | They propose a large-scale Chinese cloze test dataset ChID which studies the comprehension of idiom in Chinese. |
| Outcome: | The proposed dataset compares the performance of the proposed model with human models. |
Pedagogically Aligned Objectives Create Reliable Automatic Cloze Tests (2024.naacl-long)
Copied to clipboard
| Challenge: | a recent study has focused on the cloze training objective of Masked Language Models . distractors must be distinct and incorrect, and can be biased if the test creator is testing two versions of a text . |
| Approach: | They propose a method that jointly optimizes sets of distractors from Masked Language Models. |
| Outcome: | The proposed method has stronger correlation with teacher-created comprehension tests than state-of-the-art neural method and is more internally consistent. |
Cloze Quality Estimation for Language Assessment (2023.findings-eacl)
Copied to clipboard
| Challenge: | Cloze tests are widely used in language proficiency tests, but they suffer from low quality and low reliability. |
| Approach: | They propose a task to evaluate whether a cloze test is of sufficient "high-quality" they use a dataset that includes English clozing tests and corresponding evaluations by native English speakers. |
| Outcome: | The proposed method could contribute to the CQE task, but the task is still challenging. |
Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal (2026.acl-long)
Copied to clipboard
| Challenge: | cloze task is being replaced with LM probabilities for how predictable a word is in its context . clozen task is used to measure how predictable word is compared to unexpected words . |
| Approach: | They propose to use LM probabilities to quantify how predictable a word is . they propose to test whether LMs are better at distinguishing semantically similar words . |
| Outcome: | The results show that LM probabilities outperform cloze probabilities for the right reasons . they also show that human-like prediction is more sensitive to LM probability distinctions . |
SCDE: Sentence Cloze Dataset with High Quality Distractors From Examinations (2020.acl-main)
Copied to clipboard
| Challenge: | Experimental results show that there is a significant performance gap between advanced models (72%) and humans (87%) Cloze datasets are convenient either to be generated automatically or by annotators. |
| Approach: | They propose to use a dataset to evaluate the performance of computational models through sentence prediction. |
| Outcome: | The proposed model fills up multiple blanks in a passage from a shared candidate set with distractors designed by English teachers. |
A Corpus for Commonsense Inference in Story Cloze Test (2022.lrec-1)
Copied to clipboard
| Challenge: | Story Cloze Test (SOTA) models can achieve over 90% accuracy on predicting the last sentence, but high accuracy can be achieved by merely using surface-level features. |
| Approach: | They constructed a human-labeled and human-verified commonsense knowledge inference dataset using data from 1871 stories and three human workers labeled each story. |
| Outcome: | The proposed models can achieve 90% accuracy on predicting the last sentence, but they don't perform well on new and more challenging tasks. |
Generating and Evaluating Tests for K-12 Students with Language Model Simulations: A Case Study on Sentence Reading Efficiency (2023.emnlp-main)
Copied to clipboard
| Challenge: | Developing an educational test can be expensive and time-consuming, as each item must be written by experts and then evaluated by collecting hundreds of student responses. |
| Approach: | They propose to fine-tune large language models to simulate how previous students would have responded to unseen items to generate high-quality parallel tests. |
| Outcome: | The proposed test forms are designed to be content-equivalent and produce identical individual scores as the original test form. |
CEPOC: The Cambridge Exams Publishing Open Cloze dataset (2022.lrec-1)
Copied to clipboard
| Challenge: | This paper presents the first dataset of open cloze tests for language learners at different proficiency levels. |
| Approach: | They present the Cambridge Exams Publishing Open Cloze (CEPOC) dataset . they perform a set of experiments on three tasks: gap filling, gap prediction, and CEFR text classification. |
| Outcome: | The results of the study are promising for a number of NLP tasks. |
On the Consistency of Commonsense in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluations of commonsense for large language models focus on downstream knowledge tasks, failing to probe whether LLMs truly understand and utilize knowledge or merely memorize it. |
| Approach: | They propose to automatically construct a large benchmark named CoCo which measures LLMs’ knowledge memorization, comprehension, and application and examines the consistency between these tasks. |
| Outcome: | The proposed benchmark systematically assesses LLMs’ knowledge memorization, comprehension, and application and examines the consistency between these tasks. |