| Challenge: | Prior work on shortcut detection focused on enumerating features like unigrams or bigrams . prior work relied on post-hoc models that reveal qualitative patterns without a clear statistical interpretation . |
| Approach: | They propose to use probabilistic grammars to characterize and discover shortcuts in NLP datasets using context-free grammars and synchronous context- free grammars. |
| Outcome: | The proposed grammars reveal interesting shortcut features in a number of datasets, including simple and high-level features, and automatically identify groups of test examples on which conventional classifiers fail. |
Similar Papers
Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free Approach (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Shortcut reasoning is an irrational process of inference, which degrades the robustness of an NLP model. |
| Approach: | They propose a method to quantify the severity of shortcut reasoning by leveraging out-of-distribution data. |
| Outcome: | The proposed method quantifies the severity of the discovered shortcut reasoning using out-of-distribution data. |
Categorial Grammar Induction with Stochastic Category Selection (2024.lrec-main)
Copied to clipboard
| Challenge: | categorial grammar inducers have been used to learn from raw data, but they use shortcuts to ensure branching behavior. |
| Approach: | They propose a grammar inducer that learns from raw data and does not rely on bias terms . they show a recall-homogeneity of 0.48 on a corpus of English child-directed speech . |
| Outcome: | The proposed model achieves a recall-homogeneity of 0.48 on a corpus of English child-directed speech . |
Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) often rely on spurious correlations rather than causally relevant features to improve accuracy and generalizability. |
| Approach: | They propose a benchmark that categorizes shortcuts into occurrence, style, and concept . they aim to explore the nuanced ways shortcuts influence the performance of LMs . |
| Outcome: | The proposed benchmark categorizes shortcuts into occurrence, style, and concept . it systematically investigates models’ resilience and susceptibilities to sophisticated shortcuts . |
Supervised and Unsupervised Probing of Shortcut Learning: Case Study on the Emergence and Evolution of Syntactic Heuristics in BERT (2025.findings-acl)
Copied to clipboard
| Challenge: | Contemporary language models (LMs) rely on shortcut learning, using superficial cues that are spuriously correlated with labels. |
| Approach: | They propose to use syntactic heuristics to learn shortcuts in BERT when performing a task in Natural Language Understanding to investigate where these shortcuts emerge, how they evolve and how they impact the latent knowledge of the LM. |
| Outcome: | The proposed model rely on syntactic heuristics when performing a task in Natural Language Understanding. |
Compound Probabilistic Context-Free Grammars for Grammar Induction (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to grammar induction have resorted to manually-engineered features and auxiliary objectives to induce the desired structures. |
| Approach: | They propose a formalization of the grammar induction problem that models sentences as being generated by a compound probabilistic context free grammar. |
| Outcome: | Experiments on English and Chinese show that the proposed approach is more efficient than other methods. |
Categorial grammar induction from raw data (2023.findings-acl)
Copied to clipboard
| Challenge: | a new model for categorial grammar induction is based on raw data without part-of-speech information. |
| Approach: | They propose a grammar induction model that learns from raw data without part-of-speech information. |
| Outcome: | a new model for inducing a basic categorial grammar is developed . the model attains a recall-homogeneity of 0.33 on average, and a bias toward forward function application is added . |
Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing work identifies task-specific shortcuts via human priors or error analyses, which requires extensive expertise and efforts. |
| Approach: | They propose to automatically identify spurious correlations in NLP models at scale by using existing interpretability methods to extract tokens that significantly affect model’s decision process. |
| Outcome: | The proposed method can identify spurious correlations in NLP models at scale and mitigate these leads to more robust models in multiple applications. |
Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities in various tasks, but may rely on dataset biases as shortcuts for prediction. |
| Approach: | They propose to use a test suite to evaluate the impact of shortcuts on LLMs' performance. |
| Outcome: | The proposed test suite incorporates six shortcut types, five evaluation metrics, and four prompting strategies. |
HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizability (2021.acl-long)
Copied to clipboard
| Challenge: | Using data augmentation to fine-tune pre-trained models with task-specific data has been shown to be ineffective and redundant during fine-timing. |
| Approach: | They propose a data augmentation technique to regularize pre-trained models and encourage them to learn more generalizable features by dropping contiguous spans during training. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the GLUE benchmark and consistently exhibits superior generalization performances on out-of distribution and challenging counterexamples. |
ListOps: A Diagnostic Dataset for Latent Tree Learning (N18-4)
Copied to clipboard
| Challenge: | Existing work on latent tree learning models shows they do not learn plausible grammars . a dataset is created to study the parsing ability of such models in natural language . |
| Approach: | They propose a toy dataset to study the parsing ability of latent tree learning models . they propose 'listops' toy that has a single correct parse strategy that a system needs to learn . |
| Outcome: | The proposed model outperforms existing models on sentence understanding tasks . it can learn grammars that conform to plausible semantics and syntactic formalisms . |