| Challenge: | a large number of test instances overlap considerably with pretraining corpora, a study finds . for a number of years, models struggled to exceed chance-level performance . |
| Approach: | They analyze the effects of varying degrees of overlaps that occur between pretraining corpora and test instances in WSC-style tasks. |
| Outcome: | The WSC-Web dataset is the largest to date and has lower overlaps with current pretraining corpora. |
Similar Papers
A Surprisingly Robust Trick for the Winograd Schema Challenge (P19-1)
Copied to clipboard
| Challenge: | The Winograd Schema Challenge (WSC) dataset WSC273 and its inference counterpart WNLI are popular benchmarks for natural language understanding and commonsense reasoning. |
| Approach: | They propose to fine-tune language models on the Winograd Schema Challenge dataset WSC273 and its inference counterpart WNLI to achieve accuracies of 72.5% and 74.7%, respectively. |
| Outcome: | The proposed language models achieve 72.5% and 74.7% accuracy on the WSC273 and WNLI datasets, respectively. |
EvoGrad: A Dynamic Take on the Winograd Schema Challenge with Human Adversaries (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models excel at the Winograd Schema Challenge, but struggle with instances that feature minor alterations or rewording. |
| Approach: | They propose an open-source platform that harnesses a human-in-the-loop approach to create a dynamic dataset tailored to such altered WSC instances. |
| Outcome: | The proposed model outperforms existing models in the Winograd Schema Challenge (WSC) a human-in-the-loop approach allows for a dynamic dataset tailored to such altered instances. |
Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have boosted performance on some WS benchmarks, but the source of improvement is not clear. |
| Approach: | They propose a method that uses twin sentences for evaluation and two new baselines that account for artifacts in WS benchmarks. |
| Outcome: | The proposed evaluation method is suboptimal for the Winograd Schema . it uses twin sentences to account for commonsense reasoning abilities . |
Precise Task Formalization Matters in Winograd Schema Evaluations (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent results report a surge in performance to nearhuman levels on the Winograd Schema Challenge (WSC) however, variations in task formulation across papers and evaluations makes it hard to understand the true degree of recent progress. |
| Approach: | They propose to use a model with multiple choice to frame the task as multiple choice and reuse a pretrained language modeling head to mitigate the model's extreme sensitivity to hyperparameters. |
| Outcome: | The proposed frameworks improve the model's reasoning ability by framing the task as multiple choice and reuse of a pretrained language modeling head. |
Combining Knowledge Hunting and Neural Language Models to Solve the Winograd Schema Challenge (P19-1)
Copied to clipboard
| Challenge: | Existing methods to solve Winograd Schema Challenge use only knowledge embedded in text . this limits the performance of such models on the WSC problems. |
| Approach: | They propose to augment existing language models with a commonsense knowledge hunting module and an explicit reasoning module to extract the needed knowledge from text. |
| Outcome: | The proposed system improves on the language model based methods by 5.53% and 7.7% on the dataset. |
How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG (D19-1)
Copied to clipboard
| Challenge: | a recent study has improved the state-of-the-art on common-sense reasoning benchmarks . a san francisco-based approach to common-ense reasoning is challenging . |
| Approach: | They propose to use common-sense reasoning benchmarks to test machine learning's common-sentence inference task SWAG to test common-mind systems. |
| Outcome: | a new study shows that improved performance on common-sense reasoning benchmarks is genuine . the proposed task is more difficult than the current one, but it is more efficient than the previous one. |
WSC+: Enhancing The Winograd Schema Challenge Using Tree-of-Experts (2024.eacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel at answering WSC questions, but their ability to generate such questions remains less explored. |
| Approach: | They propose a tree-of-experts prompting method which enhances the generation of WSC instances by incorporating new 'ambiguous' and 'offensive' categories. |
| Outcome: | The proposed method enhances the generation of WSC instances (50% valid cases vs. 10% in recent methods) it extends the framework by incorporating new ‘ambiguous’ and ‘offensive’ categories, providing a deeper insight into model overconfidence and bias. |
The Sensitivity of Language Models and Humans to Winograd Schema Perturbations (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are driving recent improvements in perfromance on the Winograd Schema Challenge . a diagnostic dataset shows that these models are sensitive to linguistic perturbations that minimally affect human understanding . |
| Approach: | They propose to use a dataset to test pre-trained language models for the Winograd Schema Challenge . they show that these models are sensitive to linguistic perturbations that minimally affect human understanding . |
| Outcome: | The proposed models are sensitive to linguistic perturbations that minimally affect human understanding. |
Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models have demonstrated remarkable success in tasks like the Winograd Schema Challenge (WSC), showcasing advanced textual common-sense reasoning. |
| Approach: | They propose a framework to isolate models' ability in pronoun disambiguation from other visual processing challenges. |
| Outcome: | The proposed framework isolates the models’ ability in pronoun disambiguation from other visual processing challenges. |
Koala: An Index for Quantifying Overlaps with Pre-training Corpora (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models can be influenced by the frequency of overlap between pre-training corpora. |
| Approach: | They propose to search over large pre-training corpora using lossless compressed suffix arrays with highly efficient compression rate and search support. |
| Outcome: | Koala is a searchable index over large pre-training corpora using lossless compressed suffix arrays with highly efficient compression rate and search support. |