| Challenge: | Recent approaches to sequence to sequence learning leverage recurrence, convolution, attention or combination of recurrent and convolutional neural networks. |
| Approach: | They propose an approach that extends the self-attention mechanism to consider representations of relative positions, or distances between sequence elements. |
| Outcome: | The proposed approach yields 1.3 BLEU and 0.3 BLUE on translation tasks . it is based on a relation-aware self-attention mechanism that can generalize to arbitrary graph-labeled inputs. |
Similar Papers
The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches for positional dependencies do not satisfy all criteria for optimal position encoding. |
| Approach: | They propose a translation-invariant self-attention approach that accounts for relative position between tokens in an interpretable fashion without conventional embeddings. |
| Outcome: | The proposed model improves on regular ALBERT on GLUE tasks while adding orders of magnitude less positional parameters. |
Improve Transformer Models with Better Relative Position Embeddings (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating position embeddings are not fully utilized in NLP tasks. |
| Approach: | They propose to generalize the absolute position embedding to a generalized relative position embedded method . they also propose to use the relative embeddable method to improve the accuracy of large models . |
| Outcome: | The proposed method improves accuracy on the SQuAD1.1 dataset compared to previous methods . it can be easily adopted as a drop-in replacement for improving accuracy of large models . |
Self-Attention with Structural Position Representations (D19-1)
Copied to clipboard
| Challenge: | Experimental results show that SANs can't encode positions of input words . SAN's are currently lacking in encoding positions of words based on position-unaware "bagof-words" theory . |
| Approach: | They propose to augment SANs with structural position representations to capture latent structure of input sentence. |
| Outcome: | The proposed approach consistently outperforms the sequential representations on translation tasks. |
Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that convolutions have been successful in natural language learning. |
| Approach: | They propose a convolutional approach to construct relative position embeddings in self-attention layers and propose 'compact attention' they propose multiple ways to integrate convolutions into Transformer self- attention. |
| Outcome: | The proposed composite attention improves performance on multiple downstream tasks, replacing absolute position embeddings, and is more expressive than convolutions in NLP. |
Enhancing Machine Translation with Dependency-Aware Self-Attention (2020.acl-main)
Copied to clipboard
| Challenge: | Currently, most neural machine translation models rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism. |
| Approach: | They propose a parameter-free, dependency-aware self-attention mechanism that integrates syntactic knowledge into a Transformer model and propose 'a parameter free approach' they also propose - a novel mechanism that improves translation quality for long sentences and in low-resource scenarios. |
| Outcome: | The proposed approach improves translation quality on English-German and English-Turkish translation tasks and in low-resource scenarios. |
PermuteFormer: Efficient Relative Position Encoding for Long Sequences (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing Transformers that scale to long sequences are not compatible with relative position encoding. |
| Approach: | They propose a Performer-based model with relative position encoding that scales linearly on long sequences. |
| Outcome: | The proposed model outperforms performer on long sequences with no computational overhead and outperformed vanilla Transformer on most of the tasks. |
Understanding How Positional Encodings Work in Transformer Model (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have reported superiority of relative PEs in translation tasks. |
| Approach: | They analyze in which part of a transformer model PEs work and compare them using experiments . they find that relative PEs should be added only to query and key of attention mechanism . |
| Outcome: | The results show that relative and absolute PEs work in a transformer model, and should be added to the query and key of an attention mechanism, not to the value. |
Self-Attention with Cross-Lingual Position Representation (2020.acl-main)
Copied to clipboard
| Challenge: | Position encoding (PE) is used to preserve word order information for natural language processing tasks, generating fixed position indices for input sequences. |
| Approach: | They propose to augment SANs with cross-lingual position representations to model bilingually aware latent structure for the input sentence. |
| Outcome: | The proposed model significantly improves translation quality over baselines on EnglishGerman, JapaneseEnglish, and ChineseEnglish translation tasks. |
Recurrent Attention for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent research questions the importance of dot-product self-attention in Transformer models and shows that most attention heads learn simple positional patterns. |
| Approach: | They propose a novel mechanism to replace dot-product self-attention with a recurrent atteNtion mechanism that directly learns attention weights without token-to-token interaction. |
| Outcome: | The proposed model outperforms the Transformer model on translation tasks with fewer parameters and inference time. |
Character-Level Translation with Self-attention (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for character-level neural machine translation operate on word-level, which makes them memory inefficient because of large vocabulary sizes. |
| Approach: | They propose a transformer-based model and a novel variant that uses convolutions to combine information from nearby characters to facilitate character interactions. |
| Outcome: | The proposed model outperforms the standard transformer model and learns more robust character alignments on bilingual and multilingual translation datasets. |