Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) often rely on spurious correlations rather than causally relevant features to improve accuracy and generalizability. |
| Approach: | They propose a benchmark that categorizes shortcuts into occurrence, style, and concept . they aim to explore the nuanced ways shortcuts influence the performance of LMs . |
| Outcome: | The proposed benchmark categorizes shortcuts into occurrence, style, and concept . it systematically investigates models’ resilience and susceptibilities to sophisticated shortcuts . |
Similar Papers
Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities in various tasks, but may rely on dataset biases as shortcuts for prediction. |
| Approach: | They propose to use a test suite to evaluate the impact of shortcuts on LLMs' performance. |
| Outcome: | The proposed test suite incorporates six shortcut types, five evaluation metrics, and four prompting strategies. |
Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown great potential for in-context learning, but their robustness and performance on downstream tasks remains limited. |
| Approach: | They propose to examine the reliance of LLMs on shortcuts or spurious correlations within prompts for downstream tasks and find larger models are more likely to utilize shortcuts in prompts during inference. |
| Outcome: | The proposed model is “lazy learner” and more likely to use shortcuts in prompts during inference. |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |
Exploring and Mitigating Shortcut Learning for Generative Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have incredible instruction-following capabilities while maintaining strong task completion ability. |
| Approach: | They propose a framework to encourage LLMs to Forget Spurious correlations and Learn from In-context information. |
| Outcome: | The proposed framework can mitigate shortcut learning by forging spurious correlations and learning from in-context information. |
Investigating Ensemble Methods for Model Robustness Improvement of Text Classifiers (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to reduce model's reliance on bias features ignore the learnability of these features. |
| Approach: | They propose to reduce models' reliance on bias features by first training models with fixed low-capacity models which ignore the learnability of the bias features. |
| Outcome: | The proposed models can perform better on out-of-distribution datasets than baseline models with a more sophisticated model design. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
Measuring and Mitigating Shortcut Reliance in Language Models with Probe-Based Representation Entanglement (2026.acl-srw)
Copied to clipboard
| Challenge: | Shortcut learning remains a major obstacle to robust NLP systems. |
| Approach: | They propose to fine-tune Gemma 3 1B Instruct and Llama 3.2 1B on two synthetic sentiment shortcuts in SST-2 and one natural shortcut in MNLI based on lexical overlap. |
| Outcome: | The proposed model improves on two synthetic sentiment shortcuts and one natural shortcut in MNLI with a 99% shortcut ratio, while Gemma drops from 91.8% to 60.2%. |
Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on building dynamic benchmarks to address data contamination issues. |
| Approach: | They propose a method for identifying shortcut neurons through comparative and causal analysis to suppress shortcut neurons. |
| Outcome: | The proposed method overestimates contaminated models and is highly generalizable across benchmarks and hyperparameter settings. |
A Survey of Confidence Estimation and Calibration in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks in various domains, but they can be unreliable due to factual errors in their generations. |
| Approach: | They summarize recent advances in LLM confidence estimation and calibration and outline their main lessons learned. |
| Outcome: | The proposed methods can be used to assess the reliability of models and to calibrate them across tasks. |
How Can We Know What Language Models Know? (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent work examines knowledge contained in language models by having the LM fill in the blanks of prompts such as “Obama is a __ by profession”. |
| Approach: | They propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. |
| Outcome: | The proposed methods improve accuracy from 31.1% to 39.6% on the LAMA benchmark for extracting relational knowledge from LMs. |