| Challenge: | Pretrained language models have shown large improvements in the commonsense reasoning benchmark COPA, but recent work has identified superficial cues in benchmark datasets which are predictive of the correct answer. |
| Approach: | They propose an extension of COPA that does not suffer from easy-to-exploit single token cues and exploits them. |
| Outcome: | The proposed extension of COPA does not suffer from easy-to-exploit single token cues. |
Similar Papers
Doing Good or Doing Right? Exploring the Weakness of Commonsense Causal Reasoning Models (2021.acl-short)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) achieve surprising performance on the Choice of Plausible Alternatives (COPA) task. |
| Approach: | They propose to add a regularization loss to the existing COPA models to mitigate the problem of semantic similarity bias by adding a normalization loss. |
| Outcome: | The proposed model improves generalization ability and performs better on a challenging dataset, BCOPA-CE, which has unbiased token distribution and is more difficult for models to distinguish cause and effect. |
Are Prompt-based Models Clueless? (2022.acl-long)
Copied to clipboard
| Challenge: | Prompting has reduced the data requirement by reusing the language model head and formatting the task input to match the pre-training objective. |
| Approach: | They propose to examine whether few-shot prompt-based models exploit superficial cues by reusing the model head and formatting the input to match the pre-training objective. |
| Outcome: | The proposed models perform well on instances with superficial cues, but often outperform random accuracy on instances without superficial cuing. |
On Commonsense Cues in BERT for Solving Commonsense Tasks (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models can capture syntactic features, semantic information and factual knowledge, but structured commonsense knowledge is not captured well. |
| Approach: | They quantitatively investigate the presence of structural commonsense cues in BERT when solving commonsensense tasks and the importance of such cue for the model prediction. |
| Outcome: | The presence of commonsense knowledge is positively correlated to the model accuracy. |
Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations (D19-60)
Copied to clipboard
| Challenge: | Pretrained deep contextual representations have advanced the state-of-the-art on various commonsense NLP tasks, but we lack a concrete understanding of their capabilities. |
| Approach: | They investigate BERT's ability to encode various commonsense features in its embedding space, but are still deficient in many areas. |
| Outcome: | The proposed model improves performance on a downstream commonsense reasoning task while using minimal data. |
Learning to Learn to be Right for the Right Reasons (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent work shows that models trained on held-out data perform poorly on hard instances . previous methods have resorted to manual methods of encouraging models not to overfit to superficial cues . |
| Approach: | They propose to explicitly learn a model that does well on both easy and hard tests . they use Choice of Plausible Alternatives and Commonsense Explanation to evaluate the model . |
| Outcome: | The proposed model performs well on easy and hard tests with superficial cues but performs poorly on hard ones without superficial cuings. |
He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing models for factuality prediction are lacking for English . Traditionally, event factualism is triggered by fixed properties of lexical items . |
| Approach: | They propose a model that exploits common surface patterns that correlate with factuality labels. |
| Outcome: | The proposed model achieves the best performance on four factuality datasets. |
Pragmatic inference of scalar implicature by LLMs (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing Large Language Models (LLMs) engage in pragmatic inference of scalar implicature, such as some. |
| Approach: | They investigate how Large Language Models (LLMs) engage in pragmatic inference of scalar implicature, such as some. |
| Outcome: | The proposed models interpret some as pragmatic implicature not all in the absence of context, aligning with human language processing. |
Are Rotten Apples Edible? Challenging Commonsense Inference Ability with Exceptions (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have shown that pre-trained language models encode commonsense relational knowledge, but they are often insensitive to context, ignoring overt contextual cues such as negations. |
| Approach: | They propose a procedure that exploits generic associations in masked language models to create model-specific entailment schemas. |
| Outcome: | The proposed procedure exploits generic associations in masked language models to create model-specific Winograd-style entailment schemas. |
Contextual Embeddings: When Are They Worth It? (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, rich contextual embeddings have enabled rapid progress on benchmarks like GLUE, but require significant computational resources during pretraining and during downstream task training and inference. |
| Approach: | They empirically compare contextual embeddings with classic pretrained embedders and a random word embeddable with a simple baseline. |
| Outcome: | The proposed models perform within 5 to 10% accuracy on industry-scale data. |
Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models’ Transferability (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Using pre-trained language models, we can apply them to specialized domains such as scientific articles or clinical data. |
| Approach: | They propose to pre-train BERT models on large text corpora and use them to generalize to token sequence classification applications. |
| Outcome: | The models pre-trained on text classification tasks perform better than the models using task-specific knowledge and share non-trivial similarities. |