| Challenge: | Recent advances in NLP have created problems with the complexity of the self-attention layer. |
| Approach: | They propose to substitute standard self-attention with a local efficient one to avoid the computation of attention weights. |
| Outcome: | The proposed model matches the baseline performance and improves efficiency by skipping the computation of weights that standard attention discards. |
Similar Papers
Syntax-guided Localized Self-attention by Constituency Syntactic Distance (2022.findings-emnlp)
Copied to clipboard
Shengyuan Hou, Jushi Kai, Haotian Xue, Bingyu Zhu, Bo Yuan, Longtao Huang, Xinbing Wang, Zhouhan Lin
| Challenge: | Recent studies have shown that Transformers is implicitly learning syntactic information from data, albeit is highly dependent on the quality and scale of the training data. |
| Approach: | They propose a syntax-guided localized self-attention model that allows directly incorporating grammar structures from an external constituency parser. |
| Outcome: | The proposed model improves translation performance on a variety of datasets, from small to large datasets and with different source languages. |
When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants (2023.acl-short)
Copied to clipboard
| Challenge: | Existing models focus on improving the efficiency of self-attention, but in practice they may be slower, especially given modest input lengths that are typical of many tasks. |
| Approach: | They propose a novel local-attention variant of a self-supervised speech model that uses input length thresholds to identify bottlenecks. |
| Outcome: | The proposed model is based on a self-attention-based model with a high input length threshold. |
Enhancing Machine Translation with Dependency-Aware Self-Attention (2020.acl-main)
Copied to clipboard
| Challenge: | Currently, most neural machine translation models rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism. |
| Approach: | They propose a parameter-free, dependency-aware self-attention mechanism that integrates syntactic knowledge into a Transformer model and propose 'a parameter free approach' they also propose - a novel mechanism that improves translation quality for long sentences and in low-resource scenarios. |
| Outcome: | The proposed approach improves translation quality on English-German and English-Turkish translation tasks and in low-resource scenarios. |
Character-Level Translation with Self-attention (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for character-level neural machine translation operate on word-level, which makes them memory inefficient because of large vocabulary sizes. |
| Approach: | They propose a transformer-based model and a novel variant that uses convolutions to combine information from nearby characters to facilitate character interactions. |
| Outcome: | The proposed model outperforms the standard transformer model and learns more robust character alignments on bilingual and multilingual translation datasets. |
Theoretical Limitations of Self-Attention in Neural Sequence Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing work suggests that the computational capabilities of self-attention to model hierarchical structures are limited. |
| Approach: | They investigate the computational power of self-attention to model formal languages . they show strong theoretical limitations of self attention to model periodic finite-state languages unless the number of layers or heads increases with input length. |
| Outcome: | The proposed models can model periodic finite-state languages, nor hierarchical structure unless the number of layers or heads increases with input length. |
Does Self-Attention Need Separate Weights in Transformers? (2025.naacl-industry)
Copied to clipboard
| Challenge: | Experimental results show a 66.53% reduction in parameter size within the attention block and competitive accuracy improvements of 3.55% and 0.89% over symmetric and pairwise attention-based models, respectively. |
| Approach: | They propose a simplified approach where a single weight matrix is used for Keys, Queries, and Values instead of separate matrices for each. |
| Outcome: | The proposed approach outperforms the BERT baseline on GLUE tasks even outperforming the standard BERT model in handling noisy and out-of-domain data. |
Recurrent Attention for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent research questions the importance of dot-product self-attention in Transformer models and shows that most attention heads learn simple positional patterns. |
| Approach: | They propose a novel mechanism to replace dot-product self-attention with a recurrent atteNtion mechanism that directly learns attention weights without token-to-token interaction. |
| Outcome: | The proposed model outperforms the Transformer model on translation tasks with fewer parameters and inference time. |
Adaptive Attention Span in Transformers (P19-1)
Copied to clipboard
| Challenge: | We extend the maximum context size of a neural network called Transformer to 8k characters. |
| Approach: | They propose a self-attention mechanism that can learn its optimal attention span . this allows for models with longer context and the capability to catch longer dependencies. |
| Outcome: | The proposed model achieves state-of-the-art performance on text8 and enwiki8 using 8k characters with no loss of performance, and maintains control over memory footprint and computational time. |
Improving BERT with Syntax-aware Local Attention (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that attention-based models benefit from more focused attention over local regions. |
| Approach: | They propose a syntax-aware local attention which restrains attention over syntactically relevant words. |
| Outcome: | The proposed model performs better on all benchmark datasets, including sentence classification and sequence labeling tasks. |
Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new framework for self-attention models is proposed to address this problem . it injects *multi-hop* relationships into the attention graph, allowing for better performance . |
| Approach: | They propose a framework that injects *multi-hop* relationships through a belief propagation process. |
| Outcome: | The proposed framework helps prevent entropy collapse in deeper layers and maintains GTD at task-appropriate levels. |