Papers with shortcuts
“Will You Find These Shortcuts?” A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on faithfulness evaluation is not conclusive and does not provide a clear answer as to how different methods are to be compared. |
| Approach: | They propose a protocol for faithfulness evaluation that makes use of partially synthetic data to obtain ground truth for feature importance ranking. |
| Outcome: | The proposed method is based on partially synthetic data and is compared with lexical shortcuts on a range of datasets and LSTM models. |
Why Machine Reading Comprehension Models Learn Shortcuts? (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that many MRC models learn shortcuts to outwit benchmarks, but the performance is unsatisfactory in real-world applications. |
| Approach: | They propose to use shortcut questions to analyze learning difficulty of MRC models . they propose to analyze the learning difficulty regarding shortcut and challenging questions . |
| Outcome: | The proposed methods show that a large proportion of shortcut questions in training data make models rely on shortcut tricks excessively. |
Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) often rely on spurious correlations rather than causally relevant features to improve accuracy and generalizability. |
| Approach: | They propose a benchmark that categorizes shortcuts into occurrence, style, and concept . they aim to explore the nuanced ways shortcuts influence the performance of LMs . |
| Outcome: | The proposed benchmark categorizes shortcuts into occurrence, style, and concept . it systematically investigates models’ resilience and susceptibilities to sophisticated shortcuts . |
GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive performance across various mathematical reasoning benchmarks. |
| Approach: | They introduce an adversarial grade school math dataset and explore whether LLMs can be more robust when questions are slightly changed. |
| Outcome: | The proposed method generates and verifies each intermediate thought based on its reasoning goal and calculation result. |
Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts? (2025.findings-acl)
Copied to clipboard
| Challenge: | Latent multi-hop reasoning is a problem in Large Language Models that can develop shortcuts by encountering the head entity and answer entity in training sequences. |
| Approach: | They propose desiderata for shortcut-free evaluation of latent multi-hop reasoning ability . they exclude test queries where head and answer entities might have co-appeared . |
| Outcome: | The proposed model can latently recall and compose single-hop facts without shortcuts, but only for certain types of queries. |
Categorial Grammar Induction with Stochastic Category Selection (2024.lrec-main)
Copied to clipboard
| Challenge: | categorial grammar inducers have been used to learn from raw data, but they use shortcuts to ensure branching behavior. |
| Approach: | They propose a grammar inducer that learns from raw data and does not rely on bias terms . they show a recall-homogeneity of 0.48 on a corpus of English child-directed speech . |
| Outcome: | The proposed model achieves a recall-homogeneity of 0.48 on a corpus of English child-directed speech . |
From Generation to Selection: Findings of Converting Analogical Problem-Solving into Multiple-Choice Questions (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Abstract and Reasoning Corpus (ARC) is a benchmark designed to evaluate reasoning abilities alone by reducing the amount of prior knowledge and data required to solve the tasks. |
| Approach: | They propose a multiple-choice format suitable for assessing stages like Understand and Apply in Large Language Models (LLMs). |
| Outcome: | The proposed model supports analogical reasoning and evidence analysis, but LLMs use shortcuts in the MC-LARC format. |
Investigating Multi-Hop Factual Shortcuts in Knowledge Editing of Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent work has demonstrated the power of large language models in recalling knowledge and reasoning. |
| Approach: | They propose to erase shortcut neurons to mitigate the associated risks . 20% of the failures are attributed to shortcuts, they find . |
| Outcome: | The proposed approach reduces failures in multi-hop knowledge editing caused by shortcuts by 20% . |
Debiasing Masks: A New Framework for Shortcut Mitigation in NLU (2022.emnlp-main)
Copied to clipboard
| Challenge: | Debiasing language models from unwanted behaviors in natural language understanding datasets is a topic with increasing interest in the NLP community. |
| Approach: | They propose a method to debiase language models from unwanted behaviors in NLU tasks by identifying pruning masks that can be applied to a finetuned model. |
| Outcome: | The proposed method shows superior performance and performance over standard methods. |
Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations (2023.acl-long)
Copied to clipboard
Lucy Lu Wang, Yulia Otmakhova, Jay DeYoung, Thinh Hung Truong, Bailey Kuehl, Erin Bransom, Byron Wallace
| Challenge: | Prior work has shown that models may exploit shortcuts that are difficult to detect using standard n-gram similarity metrics such as ROUGE. |
| Approach: | They propose to use human-assessed summary quality facets and pairwise preferences to improve MDS evaluation methods. |
| Outcome: | The proposed methods improve the quality of literature review summarization models . they use human-assessed summary quality facets and pairwise preferences . |
WebDART: Dynamic Decomposition and Re-planning for Complex Web Tasks (2026.findings-acl)
Copied to clipboard
| Challenge: | Large-language-model (LLM) agents are competent at straightforward web tasks, but struggle with complex tasks. |
| Approach: | They propose a general framework that decomposes web tasks into three subtasks . they show that WebDART lifts end-to-end success rates by 13.7 percentage points . |
| Outcome: | Evaluated on WebChoreArena, WebDART lifts success rates by 13.7 percentage points over previous state-of-the-art agents. |
Guiding LLM to Fool Itself: Automatically Manipulating Machine Reading Comprehension Shortcut Triggers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent applications of LLMs in Machine Reading Comprehension systems have shown impressive results, but the use of shortcuts has emerged as a potential threat to their reliability. |
| Approach: | They propose a framework that guides an editor to add potential shortcuts-triggers to samples. |
| Outcome: | The proposed framework can edit trigger shortcuts in samples that fool LLMs . it also shows that GPT4 can be deceived by its own edits (15% drop in F1). |
Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities in various tasks, but may rely on dataset biases as shortcuts for prediction. |
| Approach: | They propose to use a test suite to evaluate the impact of shortcuts on LLMs' performance. |
| Outcome: | The proposed test suite incorporates six shortcut types, five evaluation metrics, and four prompting strategies. |
Mitigating Spurious Correlation in Natural Language Understanding with Counterfactual Inference (2022.emnlp-main)
Copied to clipboard
Can Udomcharoenchaikit, Wuttikorn Ponwitayarat, Patomporn Payoungkhamdee, Kanruethai Masuk, Weerayut Buaphet, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | Existing approaches to debias NLU models rely on superficial patterns to produce correct predictions . lexical overlap and annotation artifacts can be used to make shortcuts . |
| Approach: | They propose a causal analysis framework to help debias NLU models by defining causal relationships and utilizing counterfactual inference to mitigate bias. |
| Outcome: | The proposed framework can improve robustness across three NLU tasks while maintaining high in-distribution performance. |
Improving the robustness of NLI models with minimax training (2023.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that our method consistently outperforms other robustness enhancement techniques on out-of-distribution adversarial test sets, while maintaining high in-distance accuracy. |
| Approach: | They propose a minimax objective between a learner model being trained for the task and an auxiliary model aiming to maximize the learner's loss by up-weighting underrepresented "hard" examples with patterns that contradict the shortcuts learned from the prevailing "easy" examples. |
| Outcome: | The proposed method outperforms other robustness enhancement techniques on out-of-distribution adversarial test sets while maintaining high in-distance accuracy. |
FairFlow: Mitigating Dataset Biases through Undecided Learning for Natural Language Understanding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing debiasing frameworks can detect known dataset biases and spurious correlations in data. |
| Approach: | They propose a framework that learns to be undecided in its predictions for data samples . they propose 'contrary' objective that learn debiased and robust representations from biased views . |
| Outcome: | The proposed framework outperforms existing methods against out-of-domain and hard test samples without compromising performance. |
Can Out-of-Distribution Evaluations Uncover Reliance on Prediction Shortcuts? A Case Study in Question Answering (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work assesses models’ generalization capabilities through the lens of performance on out-of-distribution (OOD) datasets. |
| Approach: | They challenge this assumption by comparing OOD evaluations with failure modes documented in existing question-answering (QA) models. |
| Outcome: | The proposed evaluations show that the models' generalization capabilities are under-performing on out-of-distribution datasets, while others are underperforming on in-difference datasets. |
ALVIN: Active Learning Via INterpolation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that Active Learning methods ignore example groups whose prevalence may vary . supervised fine-tuning remains a critical component of model development, authors say . |
| Approach: | They propose an approach that uses interpolations to create anchors between examples . they propose to use the model to identify informative examples that counteract shortcuts . |
| Outcome: | The proposed model outperforms state-of-the-art active learning methods on six datasets . it prioritizes high-certainty instances that integrate representations from different example groups . |
Morables: A Benchmark for Assessing Abstract Moral Reasoning in LLMs with Fables (2025.emnlp-main)
Copied to clipboard
Matteo Marcuzzo, Alessandro Zangari, Andrea Albarelli, Jose Camacho-Collados, Mohammad Taher Pilehvar
| Challenge: | Literature-based benchmarks provide a compelling framework for evaluating LLMs' capacity for complex abstract reasoning and inference. |
| Approach: | They propose a novel moral reasoning benchmark built from fables and short stories that uses adversarial variants to stress-test model robustness. |
| Outcome: | The proposed model outperforms models on fables and short stories, but is susceptible to adversarial manipulation and rely on superficial patterns rather than true moral reasoning. |