| Challenge: | Story Cloze Test (SCT) is a recent framework for evaluating story comprehension and script learning. |
| Approach: | They propose to use a crowdsourcing scheme to create a new SCT dataset to overcome some of the biases discovered in the original SCT. |
| Outcome: | The proposed model performs better than the baselines on the SCT dataset, despite human-authorship biases. |
Similar Papers
A Corpus for Commonsense Inference in Story Cloze Test (2022.lrec-1)
Copied to clipboard
| Challenge: | Story Cloze Test (SOTA) models can achieve over 90% accuracy on predicting the last sentence, but high accuracy can be achieved by merely using surface-level features. |
| Approach: | They constructed a human-labeled and human-verified commonsense knowledge inference dataset using data from 1871 stories and three human workers labeled each story. |
| Outcome: | The proposed models can achieve 90% accuracy on predicting the last sentence, but they don't perform well on new and more challenging tasks. |
A Simple and Effective Approach to the Story Cloze Test (N18-2)
Copied to clipboard
| Challenge: | Existing approaches to the Cloze Test that use feature engineering to achieve high accuracy are ignoring the training set and training a model on the validation set. |
| Approach: | They propose a fully-neural approach to the Cloze Test using skip-thought embeddings of the stories in a feed-forward network that achieves close to state-of-the-art performance without any feature engineering. |
| Outcome: | The proposed approach achieves close to state-of-the-art performance on the Cloze Test without any feature engineering. |
SCDE: Sentence Cloze Dataset with High Quality Distractors From Examinations (2020.acl-main)
Copied to clipboard
| Challenge: | Experimental results show that there is a significant performance gap between advanced models (72%) and humans (87%) Cloze datasets are convenient either to be generated automatically or by annotators. |
| Approach: | They propose to use a dataset to evaluate the performance of computational models through sentence prediction. |
| Outcome: | The proposed model fills up multiple blanks in a passage from a shared candidate set with distractors designed by English teachers. |
Narrative Modeling with Memory Chains and Semantic Supervision (P18-2)
Copied to clipboard
| Challenge: | Story comprehension requires a deep semantic understanding of the narrative, making it a challenging task. |
| Approach: | They propose a method that tracks various semantic aspects with external neural memory chains . they propose to encourage each to focus on a particular semantic aspect . |
| Outcome: | The proposed method outperforms baselines on the task of story ending prediction. |
Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models? (2024.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that stories outperform rules as the expression for retrieving commonsense from LLMs, exhibiting higher generation confidence and commonsensense accuracy. |
| Approach: | They investigate the commonsense ability of large language models expressed through stories and rules to retrieve commonsensing knowledge from LLMs. |
| Outcome: | The stories outperform rules as commonsense expressions on 28 commonsensense QA datasets, exhibiting higher generation confidence and commonsence accuracy. |
MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge (L18-1)
Copied to clipboard
| Challenge: | Various approaches for script knowledge extraction and processing have been proposed in recent years. |
| Approach: | They propose a dataset to evaluate natural language understanding approaches based on commonsense knowledge. |
| Outcome: | The proposed dataset provides test cases for the broader natural language understanding community. |
A Sentence Cloze Dataset for Chinese Machine Reading Comprehension (2020.coling-main)
Copied to clipboard
| Challenge: | Using cloze-style reading comprehension, Chinese machine reading comprehension datasets are becoming more and more popular . a new task is proposed to fill the right candidate sentence into the passage with several blanks . |
| Approach: | They propose a Chinese task to fill the right candidate sentence into a passage with blanks . they build a dataset to evaluate the difficulty of the task and make fake candidates . |
| Outcome: | The proposed task fills the right candidate sentence into the passage with blanks . the proposed dataset contains over 100K blanks within over 10K passages based on Chinese narrative stories . |
ClozEx: A Task toward Generation of English Cloze Explanation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing tasks and datasets specifically designed for generating language learner explanations for cloze questions are lacking . clozing questions are used to assess language proficiency and enhance language learning . |
| Approach: | They propose a task ClozEx to generate explanations for cloze questions in LA . they use a curated dataset of clozing questions paired with explanations . |
| Outcome: | The proposed task generates fluent explanations for cloze questions in English as a second language learners. |
BBQ: A hand-built bias benchmark for question answering (2022.findings-acl)
Copied to clipboard
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, Samuel Bowman
| Challenge: | NLP models learn social biases, but little work has been done on how these biase manifest in outputs for applied tasks like question answering (QA). |
| Approach: | They propose a dataset that highlights attested social biases against people belonging to protected classes along nine social dimensions relevant for U.S. English-speaking contexts. |
| Outcome: | The proposed dataset highlights attested social biases against people belonging to protected classes along nine social dimensions relevant for U.S. English-speaking contexts. |
End-to-End Bias Mitigation by Modelling Biases in Corpora (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that strong natural language understanding models are prone to relying on unwanted dataset biases without learning the underlying task. |
| Approach: | They propose two learning strategies to train neural models that are more robust to dataset biases and transfer better to out-of-domain datasets. |
| Outcome: | The proposed methods improve robustness in all settings and transfer better to out-of-domain datasets. |