Extending LLM Context Window with Adaptive Grouped Positional Encoding: A Training-Free Method (2025.acl-long)
Copied to clipboard
| Challenge: | Existing long-context training data is scarce and requires substantial GPU resources for training. |
| Approach: | They propose a training-free plug-and-play method to enhance long-context understanding in existing large language models. |
| Outcome: | The proposed method outperforms existing LLMs on various tasks and surpasses baseline methods. |
Similar Papers
LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) experience significant performance degradation when the input exceeds the pretraining context window due to the out-of-distribution (OOD) behavior of Rotary Position Embedding (RoPE). |
| Approach: | They propose a training-free method that remaps out-of-distribution (OOD) positions into the in-distance range with fixed mapping strategies, ignoring the dynamic relationship between input length and effective context window. |
| Outcome: | Experiments on three representative LLMs across five mainstream long-context benchmarks show that the proposed method achieves significant performance improvements compared to existing methods. |
PEPE: Long-context Extension for Large Language Models via Periodic Extrapolation Positional Encodings (2025.findings-emnlp)
Copied to clipboard
Jikun Hu, Dongsheng Guo, Yuli Liu, Qingyao Ai, Lixuan Wang, Xuebing Sun, Qilei Zhang, Quan Zhou, Cheng Luo
| Challenge: | Long-context extension attempts to extend contextual window in pre-trained LLMs . primary method involves expanding initial positional encodings, disrupting positional learning . |
| Approach: | They propose a new extension strategy based on Rotary Position Embedding to extend contextual window in pre-trained large language models. |
| Outcome: | The proposed method can extend the contextual window in pre-trained large language models . expansion disrupts positional encodings learned during pre-training, authors show . |
LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models (2025.acl-long)
Copied to clipboard
Zhiyuan Hu, Yuliang Liu, Jinman Zhao, Suyuchen Wang, WangYan WangYan, Wei Shen, Qing Gu, Anh Tuan Luu, See-Kiong Ng, Zhiwei Jiang, Bryan Hooi
| Challenge: | Large language models face significant challenges in handling long-context tasks because of their limited effective context window size during pretraining, which restricts their ability to generalize over extended sequences. |
| Approach: | They propose a training strategy for extending the context window of LLMs including impactful token analysis, position index transformation, and training optimization strategies. |
| Outcome: | Experiments on three types of LLMs show that LongRecipe can utilize long sequences while requiring only 30% of the target context window size. |
E2-LLM: Efficient and Extreme Length Extension of Large Language Models (2024.findings-acl)
Copied to clipboard
Jiaheng Liu, ZhiqiBai ZhiqiBai, Yuanxing Zhang, Chenchen Zhang, YuangZh YuangZh, Ge Zhang, JiakaiWang JiakaiWang, Haoran Que, Yukang Chen, Wenbo Su, Tiezheng Ge, Jie Fu, Wenhu Chen, Bo Zheng
| Challenge: | Existing techniques for extending context capabilities in LLMs require additional training procedures and access to datasets with long context (e.g., sequences of 32K tokens). |
| Approach: | They propose a solution to extend context capabilities in Large Language Models by training a single process over a sequence of 4K tokens. |
| Outcome: | The proposed solution significantly reduces the cost of continual-pretraining or fine-tuning over short sequences and improves robustness to diverse relative positions. |
Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective (2025.coling-main)
Copied to clipboard
| Challenge: | Enabling LLMs to handle lengthy context is currently a research hotspot . a notable challenge limiting further customization is the inability of LLM to utilize context beyond pretrained length due to the inherent flaw of rotary position embedding (RoPE). |
| Approach: | They propose to extend the RoPE from an attention perspective and on two benchmarking tasks. |
| Outcome: | The proposed extension of the RoPE improves extrapolation and retrieval errors. |
Parallel Context Windows for Large Language Models (2023.acl-long)
Copied to clipboard
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham
| Challenge: | Existing efforts to address context window limitation for off-the-shelf LLMs involve training specialized architectures. |
| Approach: | They propose a method that carves a long context into chunks and restricts attention to apply only within each window. |
| Outcome: | The proposed method shows significant improvements on in-context learning tasks with diverse input and output spaces. |
Positional Overload: Positional Debiasing and Context Window Extension for Large Language Models using Set Encoding (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models typically track the order of tokens using positional encoding, which causes two significant limitations: 1. Positional Bias: When processing long text sequences, the number of token can exceed the range the model was trained on. |
| Approach: | They propose a method that allows multiple pieces of text to be encoded in the same position, eliminating positional bias entirely. |
| Outcome: | The proposed method eliminates positional bias entirely and increases the size of the input an LLM can handle. |
Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models (2025.coling-main)
Copied to clipboard
Haoran Lian, Junmin Chen, Wei Huang, Yizhe Xiong, Wenping Hu, Guiguang Ding, Hui Chen, Jianwei Niu, Zijia Lin, Fuzheng Zhang, Di Zhang
| Challenge: | Recent studies show that Large language models struggle with handling long token sequences due to limited training context size. |
| Approach: | They propose a single-stage continual pretraining method to equip LLMs with long context modeling capabilities. |
| Outcome: | The proposed method outperforms existing methods on 4 language modeling benchmarks. |
Extending Context Window of Large Language Models from a Distributional Perspective (2024.emnlp-main)
Copied to clipboard
Yingsheng Wu, Yuxuan Gu, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin
| Challenge: | Existing scaling methods for extending context window rely on empirical approaches and lack understanding of the internal distribution within RoPE resulting in suboptimal performance. |
| Approach: | They propose to optimize the context window extending task from the view of rotary angle distribution by minimizing disturbance between rotary angles to maintain consistency with the pre-training phase. |
| Outcome: | The proposed approach reduces by up to 72% of the distributional disturbance when extending LLaMA2’s context window to 8k, and reduces it by up 32% when extending to 16k. |
LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Currently, large language models (LLMs) train on short text segments due to the computational overhead quadratic in the input lengths of their Transformer architectures. |
| Approach: | They propose a method that allows LLMs pre-trained with 2K or 4K-long segments to generalize to up to 200M length inputs while retaining perplexity. |
| Outcome: | The proposed method achieves 2.7 decoding speed up and 7.5 memory saving over the original model. |