| Challenge: | Existing shortcut mitigation approaches are model-specific, difficult to tune, computationally expensive, and fail to improve learned representations. |
| Approach: | They propose to interpolate representations of majority examples to include features from intra-class minority examples with shortcut-mitigating patterns. |
| Outcome: | The proposed method improves minority generalization over ERM and state-of-the-art mitigation methods on multiple natural language understanding tasks while preserving accuracy on majority examples. |
Similar Papers
Measuring and Mitigating Shortcut Reliance in Language Models with Probe-Based Representation Entanglement (2026.acl-srw)
Copied to clipboard
| Challenge: | Shortcut learning remains a major obstacle to robust NLP systems. |
| Approach: | They propose to fine-tune Gemma 3 1B Instruct and Llama 3.2 1B on two synthetic sentiment shortcuts in SST-2 and one natural shortcut in MNLI based on lexical overlap. |
| Outcome: | The proposed model improves on two synthetic sentiment shortcuts and one natural shortcut in MNLI with a 99% shortcut ratio, while Gemma drops from 91.8% to 60.2%. |
Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU models (2021.naacl-main)
Copied to clipboard
Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxiang Gu, Tong Sun, Xia Hu
| Challenge: | Recent studies indicate that NLU models are prone to rely on shortcut features for prediction, without achieving true language understanding. |
| Approach: | They propose a shortcut mitigation framework to suppress NLU models from making overconfident predictions for samples with large shortcut degree. |
| Outcome: | The proposed framework suppresses the model from making overconfident predictions for samples with large shortcut degree. |
Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) often rely on spurious correlations rather than causally relevant features to improve accuracy and generalizability. |
| Approach: | They propose a benchmark that categorizes shortcuts into occurrence, style, and concept . they aim to explore the nuanced ways shortcuts influence the performance of LMs . |
| Outcome: | The proposed benchmark categorizes shortcuts into occurrence, style, and concept . it systematically investigates models’ resilience and susceptibilities to sophisticated shortcuts . |
ALVIN: Active Learning Via INterpolation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that Active Learning methods ignore example groups whose prevalence may vary . supervised fine-tuning remains a critical component of model development, authors say . |
| Approach: | They propose an approach that uses interpolations to create anchors between examples . they propose to use the model to identify informative examples that counteract shortcuts . |
| Outcome: | The proposed model outperforms state-of-the-art active learning methods on six datasets . it prioritizes high-certainty instances that integrate representations from different example groups . |
Mitigating Shortcuts in Language Models with Soft Label Encoding (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models rely on spurious correlations in the data for natural language understanding (NLU) tasks. |
| Approach: | They propose a framework for debiasing shortcuts and a dummy class to encode shortcuts into a model and use it to generate soft labels. |
| Outcome: | The proposed framework significantly improves out-of-distribution generalization while maintaining satisfactory in-district accuracy. |
Improving the robustness of NLI models with minimax training (2023.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that our method consistently outperforms other robustness enhancement techniques on out-of-distribution adversarial test sets, while maintaining high in-distance accuracy. |
| Approach: | They propose a minimax objective between a learner model being trained for the task and an auxiliary model aiming to maximize the learner's loss by up-weighting underrepresented "hard" examples with patterns that contradict the shortcuts learned from the prevailing "easy" examples. |
| Outcome: | The proposed method outperforms other robustness enhancement techniques on out-of-distribution adversarial test sets while maintaining high in-distance accuracy. |
CURE: Controlled Unlearning for Robust Embeddings — Mitigating Conceptual Shortcuts in Pre-Trained Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models are susceptible to spurious, concept-driven correlations that impair robustness and fairness. |
| Approach: | They propose a framework that disentangles and suppresses conceptual shortcuts while preserving essential content information. |
| Outcome: | The proposed framework improves on IMDB and Yelp datasets with minimal computational overhead. |
Mitigating Shortcut Learning via Smart Data Augmentation based on Large Language Model (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to improve shortcut learning performance are limited by manual definition of shortcuts and inherent confirmation bias during model training. |
| Approach: | They propose a method of Smart Data Augmentation based on Large Language Models to identify shortcuts and generate their anti-shortcut counterparts. |
| Outcome: | The proposed method shows an improvement of 5.61% across various natural language processing tasks. |
Exploring and Mitigating Shortcut Learning for Generative Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have incredible instruction-following capabilities while maintaining strong task completion ability. |
| Approach: | They propose a framework to encourage LLMs to Forget Spurious correlations and Learn from In-context information. |
| Outcome: | The proposed framework can mitigate shortcut learning by forging spurious correlations and learning from in-context information. |
Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities in various tasks, but may rely on dataset biases as shortcuts for prediction. |
| Approach: | They propose to use a test suite to evaluate the impact of shortcuts on LLMs' performance. |
| Outcome: | The proposed test suite incorporates six shortcut types, five evaluation metrics, and four prompting strategies. |