| Challenge: | Recent work shows that transformers with hard attention are limited in power, but hard attention is a strong assumption. |
| Approach: | They propose a generalization of hard attention that captures attention patterns in transformers with saturated attention. |
| Outcome: | The proposed generalization of hard attention more closely captures the attention patterns learnable in practical transformers. |
Similar Papers
Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity (2022.tacl-1)
Copied to clipboard
| Challenge: | averaging hard attention is used to recognize formal languages that UHAT and GUHAT cannot recognize. |
| Approach: | They analyze three formal Transformer encoders that differ in the form of their self-attention mechanism . they find that UHAT and GUHAT Transformers can only recognize formal languages in AC0 . |
| Outcome: | The proposed models can recognize languages that UHAT and GUHAT cannot . the proposed models are based on the DYCK and PARITY languages . |
On the Ability and Limitations of Transformers to Recognize Formal Languages (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on LSTMs have not revealed their ability to model syntactic properties. |
| Approach: | They propose to build a Transformers model for a subclass of counter languages and find that their learning mechanism strongly correlates with their construction. |
| Outcome: | The proposed model generalizes well on counter languages and its learned mechanism correlates with it. |
Theoretical Limitations of Self-Attention in Neural Sequence Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing work suggests that the computational capabilities of self-attention to model hierarchical structures are limited. |
| Approach: | They investigate the computational power of self-attention to model formal languages . they show strong theoretical limitations of self attention to model periodic finite-state languages unless the number of layers or heads increases with input length. |
| Outcome: | The proposed models can model periodic finite-state languages, nor hierarchical structure unless the number of layers or heads increases with input length. |
On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers (2021.findings-acl)
Copied to clipboard
Tianchu Ji, Shraddhan Jain, Michael Ferdman, Peter Milder, H. Andrew Schwartz, Niranjan Balasubramanian
| Challenge: | Recent work shows that attention can be pruned to zeros with minimal loss in accuracy. |
| Approach: | They propose a pruning technique which quantizes attention to a 3-bit format without retraining . they find that 80% of attention values can be pruned to zeros with minimal loss in accuracy . |
| Outcome: | The proposed approach produces only a few unique attention values with minimal loss in accuracy. |
The Devil in Linear Transformer (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing linear transformers suffer from performance degradations on various tasks and corpus. |
| Approach: | They propose a new linear attention that replaces scaling with a normalization to stabilize gradients and confine attention to neighbouring tokens in early layers. |
| Outcome: | The proposed model outperforms vanilla transformers on the long-range arena benchmark while being significantly more space-time efficient. |
Do Transformers Grok Succinct Algorithms? Mechanistic Evidence for Counting Circuits (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies suggest that Transformers are inherently succinct, capable of representing recursive algorithms like binary counting over exponential state spaces. |
| Approach: | They propose to bridge this gap by testing the Succinctness Hypothesis using mechanistic interpretability on a large-scale computation task. |
| Outcome: | The proposed model can represent recursive algorithms over exponential state spaces . the proposed model is able to generalize perfectly, whereas massive LSTM baselines fail completely. |
Adaptive Transformers for Learning Multimodal Representations (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing approaches for learning visiolinguistic representations with transformers are over-parametrized and require extensive training. |
| Approach: | They propose to extend attention spans, sparse, and structured dropout methods to learn more about how the network perceives the complexity of input sequences. |
| Outcome: | The proposed approaches improve on language semantics and visiolinguistic representations, but are often over-parametrized and require large amounts of computation. |
Simulating Hard Attention Using Soft Attention (2026.tacl-1)
Copied to clipboard
| Challenge: | a central element of hard attention is attention, which computes a weighted average of values from all unmasked positions. |
| Approach: | They propose transformers that can simulate hard attention by using temperature scaling and positional embeddings. |
| Outcome: | The proposed transformers can effectively focus all attention on a subset of positions. |
Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies of the computational power of recurrent neural networks reveal a hierarchy of RNN architectures, given finite-precision assumptions. |
| Approach: | They propose to use auto-regressive Transformers with linearised attention to build RNNs . they show that many well-known results for the standard Transformer directly transfer to LTs - a new approach is proposed . |
| Outcome: | The proposed extensions overcome limitations of the LT and self-referential weight matrices. |
RealFormer: Transformer Likes Residual Attention (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing techniques to create Residual Attention Layer Transformer networks outperform the canonical Transformer on a wide spectrum of tasks. |
| Approach: | They propose a technique to create Residual Attention Layer Transformer networks that outperform the canonical Transformer on a wide spectrum of tasks. |
| Outcome: | The proposed technique outperforms the canonical Transformer on a wide spectrum of tasks including Masked Language Modeling, GLUE, SQUAD, Neural Machine Translation, WikiHop, HotpotQA, Natural Questions, and OpenKP. |