Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together (N19-1)
Copied to clipboard
| Challenge: | Neural networks equipped with self-attention have parallelizable computation and the ability to capture both long-range and local dependencies. |
| Approach: | They propose a novel attention mechanism called "Multi-mask Tensorized Self-Attention" it captures pairwise and global dependencies by a compatibility function composed of dot-product and additive attentions . |
| Outcome: | The proposed model outperforms CNN-/RNN-/attention-based models on nine NLP benchmarks with compelling memory- and time-efficiency. |
Similar Papers
Convolutional Self-Attention Networks (N19-1)
Copied to clipboard
| Challenge: | Existing models of self-attention networks lack the ability to capture dependencies regardless of distance and can be enhanced with multi-head attention. |
| Approach: | They propose a convolutional self-attention network which can be enhanced by multi-head attention by allowing the model to attend to information from different representation subspaces. |
| Outcome: | The proposed model outperforms existing models on improving locality of SANs on different language pairs and model settings. |
Rethinking Self-Attention: Towards Interpretability in Neural Parsing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work shows that attention mechanisms provide arguably explainable attention distributions that can help to interpret predictions. |
| Approach: | They propose a new self-attention layer where attention heads represent labels. |
| Outcome: | The proposed model obtains state-of-the-art results on the Penn Treebank and Chinese Treebank. |
Long Sequence Modeling with Attention Tensorization: From Sequence to Tensor Learning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a lack of attention-based models for long sequences poses challenges for long-sequence modeling . attention tensorization can be used to extend context lengths with improved efficiency . tenorization enables training of LLMs with context length longer than those trained on . |
| Approach: | They propose to tensorize long input sequences into compact tenses followed by attention on each transformed dimension. |
| Outcome: | The proposed model can be used as efficient transformer backbones to extend input context length with improved memory and time efficiency. |
Multi-Level Structured Self-Attentions for Distantly Supervised Relation Extraction (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to label large-scale data are inadequate for distantly supervised relation extraction (DS-RE). |
| Approach: | They propose a multi-level structured (2-D matrix) self-attention mechanism for DS-RE using bidirectional recurrent neural networks. |
| Outcome: | The proposed framework significantly outperforms baselines on two publicly available DS-RE datasets in terms of PR curves, P@N and F1 measures. |
Leveraging Local and Global Patterns for Self-Attention Networks (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to integrate local and global information into self-attention networks have been criticized for overlooking neighboring information. |
| Approach: | They propose a hybrid attention mechanism to leverage local and global information . they use a gating scalar to integrate both sources of information based on local contexts . |
| Outcome: | The proposed approach improves on translation tasks and shows that the two types of contexts are complementary. |
Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Attention pruning techniques have been developed to identify and exploit sparseness . previous work has taken pioneering steps to discover and explain the sparsity in attention patterns . |
| Approach: | They propose a framework that observes attention patterns in a fixed dataset and generates a global sparseness mask. |
| Outcome: | The proposed approach saves 90% of computations and maintains quality of results. |
DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration (2025.findings-acl)
Copied to clipboard
| Challenge: | Long-context understanding is crucial for many NLP applications, but transformers struggle with efficiency due to quadratic complexity of self-attention. |
| Approach: | They propose a dynamic sparse attention mechanism that assigns adaptive masks at the attention-map level, preserving heterogeneous attention patterns. |
| Outcome: | The proposed method achieves high alignment with full-attention models while reducing memory and compute overhead. |
Enhancing Machine Translation with Dependency-Aware Self-Attention (2020.acl-main)
Copied to clipboard
| Challenge: | Currently, most neural machine translation models rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism. |
| Approach: | They propose a parameter-free, dependency-aware self-attention mechanism that integrates syntactic knowledge into a Transformer model and propose 'a parameter free approach' they also propose - a novel mechanism that improves translation quality for long sentences and in low-resource scenarios. |
| Outcome: | The proposed approach improves translation quality on English-German and English-Turkish translation tasks and in low-resource scenarios. |
Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new framework for self-attention models is proposed to address this problem . it injects *multi-hop* relationships into the attention graph, allowing for better performance . |
| Approach: | They propose a framework that injects *multi-hop* relationships through a belief propagation process. |
| Outcome: | The proposed framework helps prevent entropy collapse in deeper layers and maintains GTD at task-appropriate levels. |
Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures (D18-1)
Copied to clipboard
| Challenge: | Recent studies show that non-recurrent architectures outperform RNNs in neural machine translation. |
| Approach: | They hypothesize that CNNs and self-attentional networks could extract semantic features from source text. |
| Outcome: | The proposed architectures outperform RNNs on two tasks: subject-verb agreement and word sense disambiguation. |