| Challenge: | Existing approaches to the Cloze Test that use feature engineering to achieve high accuracy are ignoring the training set and training a model on the validation set. |
| Approach: | They propose a fully-neural approach to the Cloze Test using skip-thought embeddings of the stories in a feed-forward network that achieves close to state-of-the-art performance without any feature engineering. |
| Outcome: | The proposed approach achieves close to state-of-the-art performance on the Cloze Test without any feature engineering. |
Similar Papers
Tackling the Story Ending Biases in The Story Cloze Test (P18-2)
Copied to clipboard
| Challenge: | Story Cloze Test (SCT) is a recent framework for evaluating story comprehension and script learning. |
| Approach: | They propose to use a crowdsourcing scheme to create a new SCT dataset to overcome some of the biases discovered in the original SCT. |
| Outcome: | The proposed model performs better than the baselines on the SCT dataset, despite human-authorship biases. |
Narrative Modeling with Memory Chains and Semantic Supervision (P18-2)
Copied to clipboard
| Challenge: | Story comprehension requires a deep semantic understanding of the narrative, making it a challenging task. |
| Approach: | They propose a method that tracks various semantic aspects with external neural memory chains . they propose to encourage each to focus on a particular semantic aspect . |
| Outcome: | The proposed method outperforms baselines on the task of story ending prediction. |
A Corpus for Commonsense Inference in Story Cloze Test (2022.lrec-1)
Copied to clipboard
| Challenge: | Story Cloze Test (SOTA) models can achieve over 90% accuracy on predicting the last sentence, but high accuracy can be achieved by merely using surface-level features. |
| Approach: | They constructed a human-labeled and human-verified commonsense knowledge inference dataset using data from 1871 stories and three human workers labeled each story. |
| Outcome: | The proposed models can achieve 90% accuracy on predicting the last sentence, but they don't perform well on new and more challenging tasks. |
Toward Better Storylines with Sentence-Level Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Rather than modeling fluency, the sentence-level language model can focus on longer range dependencies, which are crucial for multi-sentence coherence. |
| Approach: | They propose a sentence-level language model which selects the next sentence in a story from a finite set of fluent alternatives. |
| Outcome: | The proposed model can focus on longer range dependencies, crucial for multi-sentence coherence. |
Pedagogically Aligned Objectives Create Reliable Automatic Cloze Tests (2024.naacl-long)
Copied to clipboard
| Challenge: | a recent study has focused on the cloze training objective of Masked Language Models . distractors must be distinct and incorrect, and can be biased if the test creator is testing two versions of a text . |
| Approach: | They propose a method that jointly optimizes sets of distractors from Masked Language Models. |
| Outcome: | The proposed method has stronger correlation with teacher-created comprehension tests than state-of-the-art neural method and is more internally consistent. |
Simple and Effective Semi-Supervised Question Answering (N18-2)
Copied to clipboard
| Challenge: | Existing deep learning systems for extractive Question Answering are limited and expensive to construct. |
| Approach: | They propose a semi-supervised QA system where end user specifies a set of documents and only a few labelled examples. |
| Outcome: | The proposed system achieves 50% F1 score on SQuAD and TriviaQA with very little labeled data. |
ClozEx: A Task toward Generation of English Cloze Explanation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing tasks and datasets specifically designed for generating language learner explanations for cloze questions are lacking . clozing questions are used to assess language proficiency and enhance language learning . |
| Approach: | They propose a task ClozEx to generate explanations for cloze questions in LA . they use a curated dataset of clozing questions paired with explanations . |
| Outcome: | The proposed task generates fluent explanations for cloze questions in English as a second language learners. |
Generating Reasonable and Diversified Story Ending Using Sequence to Sequence Model with Adversarial Training (C18-1)
Copied to clipboard
| Challenge: | Story generation is a challenging problem in artificial intelligence (AI) . previous work focused on learning statistical models of event sequences from large-scale text corpora . |
| Approach: | They propose to use adversarial training to generate reasonable story endings . their model includes a generator that defines the policy of generating a story ending . |
| Outcome: | The proposed model achieves better performance on the task of Story Cloze Test with an accuracy of 62.6% compared with state-of-the-art baseline methods. |
Cloze Quality Estimation for Language Assessment (2023.findings-eacl)
Copied to clipboard
| Challenge: | Cloze tests are widely used in language proficiency tests, but they suffer from low quality and low reliability. |
| Approach: | They propose a task to evaluate whether a cloze test is of sufficient "high-quality" they use a dataset that includes English clozing tests and corresponding evaluations by native English speakers. |
| Outcome: | The proposed method could contribute to the CQE task, but the task is still challenging. |
StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing automatic story evaluation methods place a premium on story lexical level coherence, deviating from human preference. |
| Approach: | They propose a novel Story Evaluation method that mimics human preference when judging a story . the model is based on a well-annotated dataset and a longformer-encoder-decoder . |
| Outcome: | The proposed method is applicable to machine-generated and human-written stories. |