| Challenge: | Recent studies suggest that relative position encodings provide better performance than absolute position coding. |
| Approach: | They propose a mechanism to encode position and segment information into Transformer models using relative position encodings. |
| Outcome: | The proposed method achieves faster training and inference time while achieving competitive performance on GLUE, XTREME and WMT benchmarks. |
Similar Papers
Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings (2025.coling-main)
Copied to clipboard
| Challenge: | Recent results suggest that positional encodings are not necessary when training decoder-only Transformer language models. |
| Approach: | They propose a causal attention mechanism that allows Transformers to store positional information without positional encodings. |
| Outcome: | The proposed model can reconstruct the positions of tokens without positional encodings. |
Improve Transformer Models with Better Relative Position Embeddings (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating position embeddings are not fully utilized in NLP tasks. |
| Approach: | They propose to generalize the absolute position embedding to a generalized relative position embedded method . they also propose to use the relative embeddable method to improve the accuracy of large models . |
| Outcome: | The proposed method improves accuracy on the SQuAD1.1 dataset compared to previous methods . it can be easily adopted as a drop-in replacement for improving accuracy of large models . |
PermuteFormer: Efficient Relative Position Encoding for Long Sequences (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing Transformers that scale to long sequences are not compatible with relative position encoding. |
| Approach: | They propose a Performer-based model with relative position encoding that scales linearly on long sequences. |
| Outcome: | The proposed model outperforms performer on long sequences with no computational overhead and outperformed vanilla Transformer on most of the tasks. |
Dynamic Position Encoding for Transformers (2022.coling-1)
Copied to clipboard
| Challenge: | In neural machine translation, the general task of translating is to reduce the input sentence into smaller units (also known as statistical phrases), select an optimal translation for each unit, and place them in the correct order. |
| Approach: | They propose a novel architecture that relies on a feed-forward backbone and self-attention mechanism to encode sequential/positional information. |
| Outcome: | The proposed architecture improves on multiple datasets in French, Italian, and German and shows that it is more efficient than the current model. |
The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches for positional dependencies do not satisfy all criteria for optimal position encoding. |
| Approach: | They propose a translation-invariant self-attention approach that accounts for relative position between tokens in an interpretable fashion without conventional embeddings. |
| Outcome: | The proposed model improves on regular ALBERT on GLUE tasks while adding orders of magnitude less positional parameters. |
Randomized Positional Encodings Boost Length Generalization of Transformers (2023.acl-short)
Copied to clipboard
Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, Joel Veness
| Challenge: | Moreover, simply training on longer sequences is inefficient due to the quadratic computation complexity of the global attention mechanism. |
| Approach: | They propose a randomized positional encoding scheme that randomly selects an ordered subset to fit the sequence’s length. |
| Outcome: | The proposed method allows Transformers to generalize to sequences of unseen length (increasing test accuracy by 12.0% on average). |
Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers (2026.findings-eacl)
Copied to clipboard
| Challenge: | Length generalization is the ability of language models to maintain performance on inputs longer than those seen during pretraining. |
| Approach: | They propose a position encoding strategy that uses random float sampling to generalize to unseen lengths. |
| Outcome: | The proposed strategy can generalize to lengths unseen during training and in benchmarks. |
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that attention heads learn simple positional patterns . |
| Approach: | They propose to replace all but one attention head of each encoder layer with simple fixed – non-learnable – attentive patterns that are solely based on position and do not require external knowledge. |
| Outcome: | The proposed model improves translation quality and improves BLEU scores by up to 3 points in low-resource scenarios. |
A Simple yet Effective Learnable Positional Encoding Method for Improving Document Transformer Model (2022.findings-aacl)
Copied to clipboard
| Challenge: | Existing document Transformers lack a robust positional encoding mechanism to indicate and embed sequential order information in documents. |
| Approach: | They propose a positional encoding method that can be pre-trained on document datasets to improve document understanding. |
| Outcome: | The proposed method outperforms baselines on document understanding tasks in form, receipt, and invoice domains and is robust and stable on noisy data with incorrect order information. |
Understanding How Positional Encodings Work in Transformer Model (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have reported superiority of relative PEs in translation tasks. |
| Approach: | They analyze in which part of a transformer model PEs work and compare them using experiments . they find that relative PEs should be added only to query and key of attention mechanism . |
| Outcome: | The results show that relative and absolute PEs work in a transformer model, and should be added to the query and key of an attention mechanism, not to the value. |