Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks (2026.acl-short)
Copied to clipboard
| Challenge: | Xiao et al., 2024) show that softmax models display an attention sink . he argues that normalization over a probability simplex must force attention to collapse onto a stable anchor to realize a default state. |
| Approach: | They show that normalization over a trigger-conditional behavior *necessarily* induces a sink in softmax self-attention models. |
| Outcome: | The proposed model can solve a task with no sink in softmax models. |
Similar Papers
A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation (2026.acl-short)
Copied to clipboard
| Challenge: | Xiao et al., 2025) show a tendency to allocate disproportionate attention mass to early (often first) positions independent of semantic content. |
| Approach: | They find that Transformers display an attention sink: disproportionate attention to the first position. |
| Outcome: | The proposed sinks are found in GPT-2–style models with learned query biases and absolute positional embeddings. |
Softpick: No Attention Sink, No Massive Activations with Rectified Softmax (2026.findings-acl)
Copied to clipboard
| Challenge: | Quantized models using softpick outperform softmax on standard benchmarks . softmax is widely used in statistics and especially in machine learning . |
| Approach: | They introduce a rectified, not sum-to-one, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations. |
| Outcome: | The proposed model outperforms softmax on benchmarks with lower bit precisions. |
Self-Adjust Softmax (2025.emnlp-main)
Copied to clipboard
Chuanyang Zheng, Yihang Gao, Guoxuan Chen, Han Shi, Jing Xiong, Xiaozhe Ren, Chao Huang, Zhenguo Li, Yu Li
| Challenge: | Usually, tokens with larger attention scores are important for the final prediction. |
| Approach: | They propose to modify softmax(z) to z softmax and its normalized variant to improve the Transformer attention mechanism by making minor adjustments to the softmax function. |
| Outcome: | The proposed model provides enhanced gradient properties compared to the vanilla softmax function. |
Simulating Hard Attention Using Soft Attention (2026.tacl-1)
Copied to clipboard
| Challenge: | a central element of hard attention is attention, which computes a weighted average of values from all unmasked positions. |
| Approach: | They propose transformers that can simulate hard attention by using temperature scaling and positional embeddings. |
| Outcome: | The proposed transformers can effectively focus all attention on a subset of positions. |
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | a strict sum-to-one constraint forces attention sinks on irrelevant tokens, while probability mass disperses as sequence lengths increase. |
| Approach: | They propose a sink-free attention mechanism that achieves ultra-sparsity and improved robustness at longer sequence lengths without the computational overhead of projection methods. |
| Outcome: | The proposed mechanism produces >99 % exact zeros and eliminates attention sinks while maintaining competitive performance on standard and long-context benchmarks. |
Sparse Attention with Linear Units (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have suggested that sparse attention mechanisms can be made more interpretable by replacing the softmax activation with its sparser variants. |
| Approach: | They propose a method to replace softmax activation with a ReLU to achieve sparsity in attention by layer normalization with either a specialized initialization or an additional gating function. |
| Outcome: | The proposed model is easy to implement and more efficient than previously proposed sparse attention mechanisms. |
Variance Sensitivity Induces Attention Entropy Collapse and Instability in Transformers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Attention-based language models rely on the softmax function to convert attention logits into probability distributions, but this process can result in attention entropy collapse. |
| Approach: | They propose to use the softmax function to re-weight attention logits to create probability distributions, but this reweighting can lead to attention entropy collapse . they find that entropic-stable attention methods can prevent entrapment and enable more stable training by controlling or insensitive to variance of attention logit variance. |
| Outcome: | The proposed methods prevent attention entropy collapse and enable more stable training. |
On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers (2021.findings-acl)
Copied to clipboard
Tianchu Ji, Shraddhan Jain, Michael Ferdman, Peter Milder, H. Andrew Schwartz, Niranjan Balasubramanian
| Challenge: | Recent work shows that attention can be pruned to zeros with minimal loss in accuracy. |
| Approach: | They propose a pruning technique which quantizes attention to a 3-bit format without retraining . they find that 80% of attention values can be pruned to zeros with minimal loss in accuracy . |
| Outcome: | The proposed approach produces only a few unique attention values with minimal loss in accuracy. |
Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have investigated the multi-head self-attention mechanism of transformers. |
| Approach: | They propose to use a human-in-the-loop pipeline to discover task-specific attention patterns and inject them into transformer models to improve their accuracy. |
| Outcome: | The proposed methods improve the performance of transformer models by incorporating predefined patterns into their attention matrices. |
Integral Transformer: Denoising Attention, Not Too Much Not Too Little (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to reduce attention noise by integrating signals from logit distributions are prone to attention noise. |
| Approach: | They propose a self-attention mechanism that integrates signals from the logit distribution to denoise attention. |
| Outcome: | The proposed model outperforms vanilla, Cog, and Differential attention variants on knowledge and reasoning benchmarks. |