Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers (2026.findings-eacl)
Copied to clipboard
| Challenge: | Length generalization is the ability of language models to maintain performance on inputs longer than those seen during pretraining. |
| Approach: | They propose a position encoding strategy that uses random float sampling to generalize to unseen lengths. |
| Outcome: | The proposed strategy can generalize to lengths unseen during training and in benchmarks. |
Similar Papers
Randomized Positional Encodings Boost Length Generalization of Transformers (2023.acl-short)
Copied to clipboard
Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, Joel Veness
| Challenge: | Moreover, simply training on longer sequences is inefficient due to the quadratic computation complexity of the global attention mechanism. |
| Approach: | They propose a randomized positional encoding scheme that randomly selects an ordered subset to fit the sequence’s length. |
| Outcome: | The proposed method allows Transformers to generalize to sequences of unseen length (increasing test accuracy by 12.0% on average). |
Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding (2024.findings-emnlp)
Copied to clipboard
Liang Zhao, Xiachong Feng, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin, Ting Liu
| Challenge: | Existing methods to enhance length extrapolation of large language models have been developed, but a systematic survey is lacking. |
| Approach: | They propose to examine the effects of positional encoding on length extrapolation. |
| Outcome: | The proposed methods improve the extrapolation of large language models, but they are still lacking a systematic survey. |
Length Generalization of Causal Transformers without Position Encoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Besides Transformers without position encodings, the success of NoPE provides a new way to overcome the challenge of generalizing to longer sentences. |
| Approach: | They propose a parameter-efficient tuning for searching attention heads’ best temperature hyper-parameters, which substantially expands NoPE’s context size. |
| Outcome: | The proposed tuning significantly expands NoPE's context size, allowing it to generalize to longer sentences with state-of-the-art generalization algorithms. |
Resonance RoPE: Improving Context Length Generalization of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated their potential across a wide spectrum of natural language processing tasks. |
| Approach: | They propose a novel approach to narrow the generalization gap in TSTL scenarios by refining the interpolation of RoPE features for OOD positions. |
| Outcome: | The proposed approach improves performance without additional online computational costs on train-short-test-long scenarios. |
Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis (2023.acl-long)
Copied to clipboard
| Challenge: | Length extrapolation allows training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. |
| Approach: | They propose a relative positional embedding design that uses longer than the training sequence to create sandwich. |
| Outcome: | The proposed model can extrapolate to L ex L tr much better than other models. |
LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) experience significant performance degradation when the input exceeds the pretraining context window due to the out-of-distribution (OOD) behavior of Rotary Position Embedding (RoPE). |
| Approach: | They propose a training-free method that remaps out-of-distribution (OOD) positions into the in-distance range with fixed mapping strategies, ignoring the dynamic relationship between input length and effective context window. |
| Outcome: | Experiments on three representative LLMs across five mainstream long-context benchmarks show that the proposed method achieves significant performance improvements compared to existing methods. |
PermuteFormer: Efficient Relative Position Encoding for Long Sequences (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing Transformers that scale to long sequences are not compatible with relative position encoding. |
| Approach: | They propose a Performer-based model with relative position encoding that scales linearly on long sequences. |
| Outcome: | The proposed model outperforms performer on long sequences with no computational overhead and outperformed vanilla Transformer on most of the tasks. |
Robust and Unbounded Length Generalization in Autoregressive Transformer-Based Text-to-Speech (2025.naacl-long)
Copied to clipboard
Eric Battenberg, RJ Skerry-Ryan, Daisy Stanton, Soroosh Mariooryad, Matt Shannon, Julian Salazar, David Teh-Hwa Kao
| Challenge: | Autoregressive (AR) Transformer-based sequence models have difficulty generalizing to sequences longer than those seen during training. |
| Approach: | They propose a system that provides cross-attention operations with relative location information. |
| Outcome: | The proposed system matches the naturalness and expressiveness of a baseline T5-based system while eliminating problems with repeated or dropped words. |
Towards More Efficient Insertion Transformer with Fractional Positional Encoding (2023.eacl-main)
Copied to clipboard
| Challenge: | Empirical studies on text generation tasks demonstrate the effectiveness of insertion-based models. |
| Approach: | They propose a reusable positional encoding scheme for insertion transformers that allows reusing representations calculated in previous steps. |
| Outcome: | Empirical studies show that the proposed model reduces the time required to generate a token and improves decoding efficiency. |
Incorporating Noisy Length Constraints into Transformer with Length-aware Positional Encodings (2020.coling-main)
Copied to clipboard
| Challenge: | Neural Machine Translation suffers from an under-translation problem due to limited modeling of output sequence lengths. |
| Approach: | They propose a method to train a Transformer model using length constraints based on positional encoding. |
| Outcome: | The proposed method outperforms a vanilla Transformer in an English-to-Japanese translation by 3.22 points . the noise injection improved robustness for length prediction errors, especially within the window size. |