| Challenge: | In neural machine translation, the general task of translating is to reduce the input sentence into smaller units (also known as statistical phrases), select an optimal translation for each unit, and place them in the correct order. |
| Approach: | They propose a novel architecture that relies on a feed-forward backbone and self-attention mechanism to encode sequential/positional information. |
| Outcome: | The proposed architecture improves on multiple datasets in French, Italian, and German and shows that it is more efficient than the current model. |
Similar Papers
Recurrent Positional Embedding for Neural Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Existing translation systems that use positional embeddings only encode static order dependencies based on discrete numerical information, which may hinder the improvement of translation capacity. |
| Approach: | They propose a recurrent positional embedding approach based on word vectors that are learned by a neural network and integrated into existing multi-head self-attention models. |
| Outcome: | The proposed approach improves translation performance over the state-of-the-art Transformer baseline in English-to-German and NIST Chinese-to English translation tasks. |
What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on pre-trained Transformers has focused on learning the meaning of positions . Embedding the position information in the self-attention mechanism is also an indispensable factor in NLP . |
| Approach: | They propose to use feature-level analysis to examine pre-trained Transformers' position embeddings . they also use empirical experiments to determine the appropriate positional encoding function . |
| Outcome: | The results of the empirical study can guide future work to choose the appropriate positional encoding function for specific tasks. |
Improve Transformer Models with Better Relative Position Embeddings (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating position embeddings are not fully utilized in NLP tasks. |
| Approach: | They propose to generalize the absolute position embedding to a generalized relative position embedded method . they also propose to use the relative embeddable method to improve the accuracy of large models . |
| Outcome: | The proposed method improves accuracy on the SQuAD1.1 dataset compared to previous methods . it can be easily adopted as a drop-in replacement for improving accuracy of large models . |
Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding (2024.findings-emnlp)
Copied to clipboard
Liang Zhao, Xiachong Feng, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin, Ting Liu
| Challenge: | Existing methods to enhance length extrapolation of large language models have been developed, but a systematic survey is lacking. |
| Approach: | They propose to examine the effects of positional encoding on length extrapolation. |
| Outcome: | The proposed methods improve the extrapolation of large language models, but they are still lacking a systematic survey. |
Towards More Efficient Insertion Transformer with Fractional Positional Encoding (2023.eacl-main)
Copied to clipboard
| Challenge: | Empirical studies on text generation tasks demonstrate the effectiveness of insertion-based models. |
| Approach: | They propose a reusable positional encoding scheme for insertion transformers that allows reusing representations calculated in previous steps. |
| Outcome: | Empirical studies show that the proposed model reduces the time required to generate a token and improves decoding efficiency. |
SHAPE: Shifted Absolute Position Embedding for Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing position representations suffer from a lack of generalization to test data with unseen lengths or high computational cost. |
| Approach: | They propose to achieve shift invariance by randomly shifting absolute positions during training by a SHAPE algorithm that is empirically comparable to its counterpart. |
| Outcome: | The proposed method outperforms existing representations on sequence-to-sequence tasks due to extrapolation, i.e., the ability to generalize to sequences that are longer than those observed during training. |
A Simple yet Effective Learnable Positional Encoding Method for Improving Document Transformer Model (2022.findings-aacl)
Copied to clipboard
| Challenge: | Existing document Transformers lack a robust positional encoding mechanism to indicate and embed sequential order information in documents. |
| Approach: | They propose a positional encoding method that can be pre-trained on document datasets to improve document understanding. |
| Outcome: | The proposed method outperforms baselines on document understanding tasks in form, receipt, and invoice domains and is robust and stable on noisy data with incorrect order information. |
Neural Machine Translation with Reordering Embeddings (P19-1)
Copied to clipboard
| Challenge: | Existing work exploits the reordering information in neural machine translation . experimental results show that the proposed methods can significantly improve the performance of the transformer translation system. |
| Approach: | They propose a reordering mechanism to learn the re ordering embedding of a word based on contextual information and stack them together with self-attention networks to learn sentence representation for machine translation. |
| Outcome: | The proposed method improves translation performance on English-to-German, NIST Chinese-to English, and WAT Japanese-toEnglish translation tasks. |
The Impact of Positional Encodings on Multilingual Compression (2021.emnlp-main)
Copied to clipboard
| Challenge: | Several modifications have been proposed to improve monolingual language models, but none of them result in better multilingual models. |
| Approach: | They propose to add positional encodings to token embeddings to preserve word-order information in a non-autoregressive setting. |
| Outcome: | The proposed modifications tend to improve monolingual models, but none improve multilingual models. |
Beyond Position: the emergence of wavelet-like properties in Transformers (2025.acl-long)
Copied to clipboard
| Challenge: | Despite its widespread adoption, theoretical limitations in positional encodings are resolved by developing emergent, wavelet-like processing strategies. |
| Approach: | They propose to use Rotary Position Embeddings to develop emergent, wavelet-like properties that compensate for the positional encoding’s theoretical limitations. |
| Outcome: | The attention heads evolve to implement multi-resolution processing analogous to wavelet transforms. |