| Challenge: | The Winograd Schema Challenge (WSC) dataset WSC273 and its inference counterpart WNLI are popular benchmarks for natural language understanding and commonsense reasoning. |
| Approach: | They propose to fine-tune language models on the Winograd Schema Challenge dataset WSC273 and its inference counterpart WNLI to achieve accuracies of 72.5% and 74.7%, respectively. |
| Outcome: | The proposed language models achieve 72.5% and 74.7% accuracy on the WSC273 and WNLI datasets, respectively. |
Similar Papers
WSC+: Enhancing The Winograd Schema Challenge Using Tree-of-Experts (2024.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel at answering WSC questions, but their ability to generate such questions remains less explored. |
| Approach: | They propose a tree-of-experts prompting method which enhances the generation of WSC instances by incorporating new 'ambiguous' and 'offensive' categories. |
| Outcome: | The proposed method enhances the generation of WSC instances (50% valid cases vs. 10% in recent methods) it extends the framework by incorporating new ‘ambiguous’ and ‘offensive’ categories, providing a deeper insight into model overconfidence and bias. |
An Analysis of Dataset Overlap on Winograd-Style Tasks (2020.coling-main)
Copied to clipboard
| Challenge: | a large number of test instances overlap considerably with pretraining corpora, a study finds . for a number of years, models struggled to exceed chance-level performance . |
| Approach: | They analyze the effects of varying degrees of overlaps that occur between pretraining corpora and test instances in WSC-style tasks. |
| Outcome: | The WSC-Web dataset is the largest to date and has lower overlaps with current pretraining corpora. |
EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversaries (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models excel at the Winograd Schema Challenge, but struggle with instances that feature minor alterations or rewording. |
| Approach: | They propose an open-source platform that harnesses a human-in-the-loop approach to create a dynamic dataset tailored to such altered WSC instances. |
| Outcome: | The proposed model outperforms existing models in the Winograd Schema Challenge (WSC) a human-in-the-loop approach allows for a dynamic dataset tailored to such altered instances. |
Combining Knowledge Hunting and Neural Language Models to Solve the Winograd Schema Challenge (P19-1)
Copied to clipboard
| Challenge: | Existing methods to solve Winograd Schema Challenge use only knowledge embedded in text . this limits the performance of such models on the WSC problems. |
| Approach: | They propose to augment existing language models with a commonsense knowledge hunting module and an explicit reasoning module to extract the needed knowledge from text. |
| Outcome: | The proposed system improves on the language model based methods by 5.53% and 7.7% on the dataset. |
Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution (2021.emnlp-main)
Copied to clipboard
| Challenge: | Winograd schemas are well-established tools for evaluating coreference resolution and commonsense reasoning capabilities of computational models. |
| Approach: | They present a dataset of German, French, and Russian schemas aligned with their English counterparts. |
| Outcome: | The proposed model improves in English and German, while the model improve in other languages. |
WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge (2020.acl-main)
Copied to clipboard
| Challenge: | a new task is developed to categorize essential commonsense knowledge for the Winograd Schema Challenge . a previous work focused on the classification of commonsensical knowledge by pre-trained models . |
| Approach: | They propose a task that categorizes essential commonsense knowledge for the Winograd Schema Challenge . they leverage the collected reasons to identify plausible reasons from wrong ones . |
| Outcome: | The proposed task aims to shed light on commonsense knowledge needed for better reasoning . it is based on the collected reasons and a new task called WinoWhy . |
Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have boosted performance on some WS benchmarks, but the source of improvement is not clear. |
| Approach: | They propose a method that uses twin sentences for evaluation and two new baselines that account for artifacts in WS benchmarks. |
| Outcome: | The proposed evaluation method is suboptimal for the Winograd Schema . it uses twin sentences to account for commonsense reasoning abilities . |
Precise Task Formalization Matters in Winograd Schema Evaluations (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent results report a surge in performance to nearhuman levels on the Winograd Schema Challenge (WSC) however, variations in task formulation across papers and evaluations makes it hard to understand the true degree of recent progress. |
| Approach: | They propose to use a model with multiple choice to frame the task as multiple choice and reuse a pretrained language modeling head to mitigate the model's extreme sensitivity to hyperparameters. |
| Outcome: | The proposed frameworks improve the model's reasoning ability by framing the task as multiple choice and reuse of a pretrained language modeling head. |
Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models have demonstrated remarkable success in tasks like the Winograd Schema Challenge (WSC), showcasing advanced textual common-sense reasoning. |
| Approach: | They propose a framework to isolate models' ability in pronoun disambiguation from other visual processing challenges. |
| Outcome: | The proposed framework isolates the models’ ability in pronoun disambiguation from other visual processing challenges. |
Concept-Reversed Winograd Schema Challenge: Evaluating and Improving Robust Reasoning in Large Language Models via Abstraction (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent research has revealed that Large Language Models (LLMs) often experience issues with hallucinations and unreliable reasoning due to semantic associations and superficial logical chains. |
| Approach: | They propose a concept-reversed Winograd Schema Challenge dataset to evaluate the robustness of Large Language Models (LLMs) they propose Abstraction-of-Thought (AoT) method for recovering adversarial cases to normal cases using conceptual abstraction to improve LLMs’ robustness and consistency in reasoning. |
| Outcome: | The proposed method improves LLMs’ robustness and consistency in reasoning under adversarial and long-tail scenarios. |