Softpick: No Attention Sink, No Massive Activations with Rectified Softmax (2026.findings-acl)
Copied to clipboard
| Challenge: | Quantized models using softpick outperform softmax on standard benchmarks . softmax is widely used in statistics and especially in machine learning . |
| Approach: | They introduce a rectified, not sum-to-one, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations. |
| Outcome: | The proposed model outperforms softmax on benchmarks with lower bit precisions. |
Similar Papers
Self-Adjust Softmax (2025.emnlp-main)
Copied to clipboard
Chuanyang Zheng, Yihang Gao, Guoxuan Chen, Han Shi, Jing Xiong, Xiaozhe Ren, Chao Huang, Zhenguo Li, Yu Li
| Challenge: | Usually, tokens with larger attention scores are important for the final prediction. |
| Approach: | They propose to modify softmax(z) to z softmax and its normalized variant to improve the Transformer attention mechanism by making minor adjustments to the softmax function. |
| Outcome: | The proposed model provides enhanced gradient properties compared to the vanilla softmax function. |
Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks (2026.acl-short)
Copied to clipboard
| Challenge: | Xiao et al., 2024) show that softmax models display an attention sink . he argues that normalization over a probability simplex must force attention to collapse onto a stable anchor to realize a default state. |
| Approach: | They show that normalization over a trigger-conditional behavior *necessarily* induces a sink in softmax self-attention models. |
| Outcome: | The proposed model can solve a task with no sink in softmax models. |
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | a strict sum-to-one constraint forces attention sinks on irrelevant tokens, while probability mass disperses as sequence lengths increase. |
| Approach: | They propose a sink-free attention mechanism that achieves ultra-sparsity and improved robustness at longer sequence lengths without the computational overhead of projection methods. |
| Outcome: | The proposed mechanism produces >99 % exact zeros and eliminates attention sinks while maintaining competitive performance on standard and long-context benchmarks. |
Integral Transformer: Denoising Attention, Not Too Much Not Too Little (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to reduce attention noise by integrating signals from logit distributions are prone to attention noise. |
| Approach: | They propose a self-attention mechanism that integrates signals from the logit distribution to denoise attention. |
| Outcome: | The proposed model outperforms vanilla, Cog, and Differential attention variants on knowledge and reasoning benchmarks. |
Sparse Attention with Linear Units (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have suggested that sparse attention mechanisms can be made more interpretable by replacing the softmax activation with its sparser variants. |
| Approach: | They propose a method to replace softmax activation with a ReLU to achieve sparsity in attention by layer normalization with either a specialized initialization or an additional gating function. |
| Outcome: | The proposed model is easy to implement and more efficient than previously proposed sparse attention mechanisms. |
Simulating Hard Attention Using Soft Attention (2026.tacl-1)
Copied to clipboard
| Challenge: | a central element of hard attention is attention, which computes a weighted average of values from all unmasked positions. |
| Approach: | They propose transformers that can simulate hard attention by using temperature scaling and positional embeddings. |
| Outcome: | The proposed transformers can effectively focus all attention on a subset of positions. |
Sparsifying Transformer Models with Trainable Representation Pooling (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to sparsify attention in the Transformer model are based on quadratic memory complexity and a lack of information for each word. |
| Approach: | They propose a method to sparsify attention in a Transformer model by learning to select the most-informative token representations during the training process. |
| Outcome: | The proposed model performs better than the current SOTA model while being 1.8 faster during training, 4.5 faster inference and 13 more efficient in the decoder. |
On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers (2021.findings-acl)
Copied to clipboard
Tianchu Ji, Shraddhan Jain, Michael Ferdman, Peter Milder, H. Andrew Schwartz, Niranjan Balasubramanian
| Challenge: | Recent work shows that attention can be pruned to zeros with minimal loss in accuracy. |
| Approach: | They propose a pruning technique which quantizes attention to a 3-bit format without retraining . they find that 80% of attention values can be pruned to zeros with minimal loss in accuracy . |
| Outcome: | The proposed approach produces only a few unique attention values with minimal loss in accuracy. |
A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation (2026.acl-short)
Copied to clipboard
| Challenge: | Xiao et al., 2025) show a tendency to allocate disproportionate attention mass to early (often first) positions independent of semantic content. |
| Approach: | They find that Transformers display an attention sink: disproportionate attention to the first position. |
| Outcome: | The proposed sinks are found in GPT-2–style models with learned query biases and absolute positional embeddings. |
Learning Adaptive Axis Attentions in Fine-tuning: Beyond Fixed Sparse Attention Patterns (2022.findings-acl)
Copied to clipboard
Zihan Wang, Jiuxiang Gu, Jason Kuen, Handong Zhao, Vlad Morariu, Ruiyi Zhang, Ani Nenkova, Tong Sun, Jingbo Shang
| Challenge: | Adaptive Axis Attention learns different attention patterns for each task and model layer . sparse attention patterns do not improve the run time of the models but they reduce model memory requirements . |
| Approach: | They propose a method that learns different attention patterns for each Transformer layer . they propose 'adaptive axis attention' method that identifies important tokens . |
| Outcome: | The proposed method does not require pre-training to accommodate sparse attention patterns. |