Theoretical Limitations of Self-Attention in Neural Sequence Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing work suggests that the computational capabilities of self-attention to model hierarchical structures are limited. |
| Approach: | They investigate the computational power of self-attention to model formal languages . they show strong theoretical limitations of self attention to model periodic finite-state languages unless the number of layers or heads increases with input length. |
| Outcome: | The proposed models can model periodic finite-state languages, nor hierarchical structure unless the number of layers or heads increases with input length. |
Similar Papers
Self-Attention Networks Can Process Bounded Hierarchical Languages (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models that can process formal languages with hierarchical structure are limited in their performance. |
| Approach: | They propose to use a subset of Dyck-k with depth bounded by D to train self-attention networks. |
| Outcome: | The proposed model can process Dyck-(k, D) with depth bounded by D, which better captures the hierarchical structure of natural language. |
Overcoming a Theoretical Limitation of Self-Attention (2022.acl-long)
Copied to clipboard
| Challenge: | Hahn shows that for languages where acceptance depends on a single input symbol, a transformer’s classification decisions get closer and closer to random guessing as input strings get longer and longer. |
| Approach: | They propose a transformer that recognizes PARITY with perfect accuracy and a model that uses layer normalization to bring the cross-entropy of both models arbitrarily close to zero. |
| Outcome: | The proposed model can accept and reject strings with perfect accuracy and bring cross-entropy close to zero when they need to focus on a single position. |
Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies of the computational power of recurrent neural networks reveal a hierarchy of RNN architectures, given finite-precision assumptions. |
| Approach: | They propose to use auto-regressive Transformers with linearised attention to build RNNs . they show that many well-known results for the standard Transformer directly transfer to LTs - a new approach is proposed . |
| Outcome: | The proposed extensions overcome limitations of the LT and self-referential weight matrices. |
On the Ability and Limitations of Transformers to Recognize Formal Languages (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on LSTMs have not revealed their ability to model syntactic properties. |
| Approach: | They propose to build a Transformers model for a subclass of counter languages and find that their learning mechanism strongly correlates with their construction. |
| Outcome: | The proposed model generalizes well on counter languages and its learned mechanism correlates with it. |
On the Locality of Attention in Direct Speech Translation (2022.acl-srw)
Copied to clipboard
| Challenge: | Recent advances in NLP have created problems with the complexity of the self-attention layer. |
| Approach: | They propose to substitute standard self-attention with a local efficient one to avoid the computation of attention weights. |
| Outcome: | The proposed model matches the baseline performance and improves efficiency by skipping the computation of weights that standard attention discards. |
How Much Attention Do You Need? A Granular Analysis of Neural Machine Translation Architectures (P18-1)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) has been replaced by convolutional or self-attentional approaches. |
| Approach: | They propose an architecture definition language that allows for a flexible combination of common building blocks. |
| Outcome: | The proposed architectures can bring recurrent and convolutional models close to the Transformer architecture, but not using self-attention. |
Saturated Transformers are Constant-Depth Threshold Circuits (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent work shows that transformers with hard attention are limited in power, but hard attention is a strong assumption. |
| Approach: | They propose a generalization of hard attention that captures attention patterns in transformers with saturated attention. |
| Outcome: | The proposed generalization of hard attention more closely captures the attention patterns learnable in practical transformers. |
Revealing the Dark Secrets of BERT (D19-1)
Copied to clipboard
| Challenge: | Existing models of BERT-based learning systems are lacking specific mechanisms that contribute to its success. |
| Approach: | They propose to use GLUE tasks to analyze the interpretation of self-attention, which is one of the underlying components of BERT. |
| Outcome: | The proposed model outperforms the regular model on GLUE tasks by disabling attention in certain heads. |
Anisotropy Is Inherent to Self-Attention in Transformers (2024.eacl-long)
Copied to clipboard
| Challenge: | despite their success, Transformers models suffer from a representation degeneration problem . anisotropy is a property of hidden representations that makes them unexpectedly close to each other . |
| Approach: | They investigate the representation degeneration problem in a self-supervised learning model based on Transformers . anisotropy is a property of hidden representations which makes them unexpectedly close to each other . |
| Outcome: | The representation degeneration problem is a phenomenon widely observed among self-supervised learning methods based on Transformers. |
Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that Transformers can recognize hierarchical structures without a specific positional encoding. |
| Approach: | They show that Transformers can generate hierarchical languages without a positional encoding . they also suggest that explicit positional encoders might have a detrimental effect on generalization . |
| Outcome: | The proposed model can generate hierarchical languages with respect to model size without encoding . |