| Challenge: | Neural networks are surprisingly good at interpolating, but they are often unable to extrapolate patterns beyond the seen data. |
| Approach: | They propose to use a special type of extrapolation for natural language processing to generalize to sequences that are longer than the training ones. |
| Outcome: | The proposed model is more likely to extrapolate than models with common attention mechanisms. |
Similar Papers
How Well Can a Long Sequence Model Model Long Sequences? Comparing Architectural Inductive Biases on Long-Context Abilities (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in system engineering and model design have enabled extended context models. |
| Approach: | They propose to scale up models that are purported to support extended contexts . they show that recurrent models still suffer in the same settings as long-context LLMs if attention is given to them . |
| Outcome: | The proposed models can extend to infinite sequence length, but they suffer in the same settings as long-context models with attention. |
Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective (2025.coling-main)
Copied to clipboard
| Challenge: | Enabling LLMs to handle lengthy context is currently a research hotspot . a notable challenge limiting further customization is the inability of LLM to utilize context beyond pretrained length due to the inherent flaw of rotary position embedding (RoPE). |
| Approach: | They propose to extend the RoPE from an attention perspective and on two benchmarking tasks. |
| Outcome: | The proposed extension of the RoPE improves extrapolation and retrieval errors. |
Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding (2024.findings-emnlp)
Copied to clipboard
Liang Zhao, Xiachong Feng, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin, Ting Liu
| Challenge: | Existing methods to enhance length extrapolation of large language models have been developed, but a systematic survey is lacking. |
| Approach: | They propose to examine the effects of positional encoding on length extrapolation. |
| Outcome: | The proposed methods improve the extrapolation of large language models, but they are still lacking a systematic survey. |
Beyond Paragraphs: NLP for Long Sequences (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we will introduce document-level representation learning techniques . document-based learning is challenging due to the limited sequence length of many models . |
| Approach: | They will provide an overview of established long sequence NLP techniques and discuss memory-saving methods that are key to processing long sequences. |
| Outcome: | The tutorial will introduce the latest and ongoing techniques for document-level representation learning. |
How Do Neural Sequence Models Generalize? Local and Global Cues for Out-of-Distribution Prediction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Using RNN and transformer language models, we show consistent generalization in out-of-distribution contexts. |
| Approach: | They propose two idealized models of generalization in next-word prediction . they show that neural language models interpolate between these two forms of generalisation . |
| Outcome: | The proposed models exhibit consistent generalization in out-of-distribution contexts. |
Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference (D18-1)
Copied to clipboard
| Challenge: | In this paper, we examine the behavior of deep learning models in their intermediate layers . saliency determines what is critical for the final decision of a deep model . |
| Approach: | They propose to interpret the intermediate layers of deep models by visualizing the saliency of attention and LSTM gating signals. |
| Outcome: | The proposed methods reveal interesting insights and identify critical information contributing to the model decisions. |
Simple Local Attentions Remain Competitive for Long-Context Tasks (2022.naacl-main)
Copied to clipboard
Wenhan Xiong, Barlas Oguz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Scott Yih, Yashar Mehdad
| Challenge: | Existing models for NLP tasks require long text sequences beyond the length limit of pretrained models. |
| Approach: | They propose to pretrain large-size NLP models using the same long-doc corpus and fine tune them for real-world long-context tasks. |
| Outcome: | The proposed models can perform better under standard pretraining paradigms than longformer and Longformer. |
A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing solutions to problem of positional out-of-distribution (O.O.D.) are inefficient, redundant, and lack local positional information. |
| Approach: | They propose a training-free method that greedily reuses pretrained positional intervals and interpolates attention logits to eliminate outliers. |
| Outcome: | The proposed method achieves stable and superior performance across long-context tasks without requiring input-length-specific tuning. |
Extending Input Contexts of Language Models through Training on Segmented Sequences (2024.findings-naacl)
Copied to clipboard
| Challenge: | Effectively training languages models on long sequences poses many technical challenges. |
| Approach: | They propose a method for extending positional embeddings by sub-sampling segments from long inputs while maintaining their original position. |
| Outcome: | The proposed method extends the input con-text size of pretrained models without any changes in the model's memory and memory costs. |
A Length-Extrapolatable Transformer (2023.acl-long)
Copied to clipboard
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, Furu Wei
| Challenge: | Existing Transformers can only deal with the in-distribution size of inputs. |
| Approach: | They propose a relative position embedding to explicitly maximize attention resolution . they also use blockwise causal attention during inference for better resolution a . |
| Outcome: | The proposed model achieves strong performance in interpolation and extrapolation settings. |