| Challenge: | Attention-based models are successful when trained on large amounts of data. |
| Approach: | They propose an approach to map human-annotated rationales to high-performing attention and use this to guide models trained in low-resource scenarios. |
| Outcome: | The proposed model yields over 15% error reduction on benchmark datasets. |
Similar Papers
BERT, are you paying attention? Attention regularization with human-annotated rationales (2026.eacl-long)
Copied to clipboard
| Challenge: | Attention regularisation aims to supervise the attention patterns in language models like BERT. |
| Approach: | They compare regularisation on human rationales with random tokens to find that human-annotated rationale is better at reducing model sensitivity to spurious correlations. |
| Outcome: | The proposed regularisation method improves model performance and model robustness, but not with human-annotated rationales. |
Attention Is (not) All You Need for Commonsense Reasoning (P19-1)
Copied to clipboard
| Challenge: | Recent language models such as word2vec have produced impressive results on various tasks such as question-answering and natural language inference. |
| Approach: | They propose a simple re-implementation of BERT for commonsense reasoning . they propose to use attention-guided reasoning to solve the Pronoun Disambiguation Problem . |
| Outcome: | The proposed model outperforms the state-of-the-art on several language understanding benchmarks while outperforming the existing models by a margin. |
Less is More: Attention Supervision with Counterfactuals for Text Classification (2020.emnlp-main)
Copied to clipboard
| Challenge: | Specifically, we explore the advantage of counterfactual reasoning, over associative reasoning . Adding human supervision to attention has been shown to improve model predictions and explanations . |
| Approach: | They propose to use machine-augmented human attention supervision to enhance model quality. |
| Outcome: | The proposed method is more effective than existing methods requiring higher annotation cost . the proposed method can be trained to generate similar attention to human supervision . |
Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Attention pruning techniques have been developed to identify and exploit sparseness . previous work has taken pioneering steps to discover and explain the sparsity in attention patterns . |
| Approach: | They propose a framework that observes attention patterns in a fixed dataset and generates a global sparseness mask. |
| Outcome: | The proposed approach saves 90% of computations and maintains quality of results. |
Learning to Deceive with Attention-Based Explanations (2020.acl-main)
Copied to clipboard
| Challenge: | Attention mechanisms are ubiquitous components in neural network architectures and are often claimed to confer interpretability. |
| Approach: | They propose a method for training models to produce deceptive attention masks by combining weights assigned to designated impermissible tokens with a weighted sum. |
| Outcome: | The proposed method reduces the weight assigned to designated impermissible tokens while still using them across multiple models and tasks. |
What to Learn, and How: Toward Effective Learning from Rationales (2022.findings-acl)
Copied to clipboard
| Challenge: | Increasing interest in learning from rationales has led to the use of human-annotated explanations to inject useful inductive biases into models. |
| Approach: | They propose several novel loss functions and learning strategies to exploit human rationales to augment model prediction accuracy. |
| Outcome: | The proposed learning strategies improve on three datasets with human rationales and show that they are more efficient than baselines. |
IAM: Efficient Inference through Attention Mapping between Different-scale LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a challenge due to their internal reasoning processes. |
| Approach: | They propose an algorithm that can optimize attention matrices by performing attention mapping between small and large LLMs. |
| Outcome: | The proposed framework can reduce KV cache usage by 22.1% and accelerate prefill by 15% without sacrificing performance. |
Attention Optimization for Abstractive Document Summarization (D19-1)
Copied to clipboard
| Challenge: | Abstractive summarization models require attention to reproduce the most salient information. |
| Approach: | They propose to use local and global variances to augment the vanilla attention model to reproduce the most salient information and avoid repetitions. |
| Outcome: | The proposed attention refinement unit can reproduce the most salient information and avoid repetitions on CNN/Daily Mail dataset. |
Exploring Attention Attractors in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies have suggested that attention attractors function as "summary tokens" while others speculate that tokens with weaker semantics attract high attention, they act as attention sinks that offload excessive attention. |
| Approach: | They examine attention attractors, tokens that draw significantly high attention, in large language models. |
| Outcome: | The proposed models are able to capture long-range dependencies within a given context. |
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Sparse attention is a promising strategy to extend long-context capabilities in LLMs . but its efficiency–accuracy trade-offs remain unclear due to the lack of comprehensive evaluation . |
| Approach: | They evaluate sparse attention methods across multiple model families and sizes . they find larger sparser models outperform smaller dense ones at equivalent cost . |
| Outcome: | The proposed methods outperform smaller sparse models at equivalent cost and improve the Pareto frontier. |