LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models

Challenge:	Large language models face significant challenges in handling long-context tasks because of their limited effective context window size during pretraining, which restricts their ability to generalize over extended sequences.
Approach:	They propose a training strategy for extending the context window of LLMs including impactful token analysis, position index transformation, and training optimization strategies.
Outcome:	Experiments on three types of LLMs show that LongRecipe can utilize long sequences while requiring only 30% of the target context window size.

Similar Papers

Overview

Challenge

Approach

Outcome

Effective Long-Context Scaling of Foundation Models (2024.naacl-long)

Copied to clipboard

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, Hao Ma

Language Models

Challenge:	Large language models (LLMs) are rapidly deployed and continue to evolve through scaling.
Approach:	They propose a method to train strong long-context LLMs that are capable of utilizing massive context windows of up to 32,000 tokens.
Outcome:	The proposed model can surpass gpt-3.5-turbo-16k's overall performance on long-context benchmarks with a cost-effective instruction tuning procedure that is free of expensive annotations.

E2-LLM: Efficient and Extreme Length Extension of Large Language Models (2024.findings-acl)

Copied to clipboard

Jiaheng Liu, ZhiqiBai ZhiqiBai, Yuanxing Zhang, Chenchen Zhang, YuangZh YuangZh, Ge Zhang, JiakaiWang JiakaiWang, Haoran Que, Yukang Chen, Wenbo Su, Tiezheng Ge, Jie Fu, Wenhu Chen, Bo Zheng

Language Models

Challenge:	Existing techniques for extending context capabilities in LLMs require additional training procedures and access to datasets with long context (e.g., sequences of 32K tokens).
Approach:	They propose a solution to extend context capabilities in Large Language Models by training a single process over a sequence of 4K tokens.
Outcome:	The proposed solution significantly reduces the cost of continual-pretraining or fine-tuning over short sequences and improves robustness to diverse relative positions.

How to Train Long-Context Language Models (Effectively) (2025.acl-long)

Copied to clipboard

Tianyu Gao, Alexander Wettig, Howard Yen, Danqi Chen

Language Models

Challenge:	a new study shows that language models can process extremely long contexts with minimal training.
Approach:	They use supervised fine-tuning and continued training to evaluate a language model's long-context capabilities.
Outcome:	The proposed model outperforms Llama-3.1-8B-Instruct on most long-context tasks . the model can process 512K tokens, one of the longest context windows of LMs .

LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models (2024.naacl-long)

Copied to clipboard

Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, Sinong Wang

Language Models

Challenge:	Currently, large language models (LLMs) train on short text segments due to the computational overhead quadratic in the input lengths of their Transformer architectures.
Approach:	They propose a method that allows LLMs pre-trained with 2K or 4K-long segments to generalize to up to 200M length inputs while retaining perplexity.
Outcome:	The proposed method achieves 2.7 decoding speed up and 7.5 memory saving over the original model.

From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models (2026.findings-acl)

Copied to clipboard

Chejian Xu, Wei Ping, Peng Xu, Zihan Liu, Boxin Wang, Mohammad Shoeybi, Bo Li, Bryan Catanzaro

Language Models

Challenge:	Long-context capabilities are essential for document and video understanding, in-contact learning, and inference-time scaling.
Approach:	They propose an efficient training recipe for building ultra-long context LLMs from aligned instruct model, pushing the boundaries of context lengths from 128K to 1M, 2M, and 4M tokens.
Outcome:	The proposed model extends the context window while maintaining short context capabilities while maintaining the performance of the existing model.

Extending LLM Context Window with Adaptive Grouped Positional Encoding: A Training-Free Method (2025.acl-long)

Copied to clipboard

Xinhao Xu, Jiaxin Li, Hui Chen, Zijia Lin, Jungong Han, Guiguang Ding

Semantic Text Processing

Challenge:	Existing long-context training data is scarce and requires substantial GPU resources for training.
Approach:	They propose a training-free plug-and-play method to enhance long-context understanding in existing large language models.
Outcome:	The proposed method outperforms existing LLMs on various tasks and surpasses baseline methods.

LongLeader: A Comprehensive Leaderboard for Large Language Models in Long-context Scenarios (2025.naacl-long)

Copied to clipboard

Pei Chen, Hongye Jin, Cheng-Che Lee, Rulin Shao, Jingfeng Yang, Mingyu Zhao, Zhaoyu Zhang, Qin Lu, Kaiwen Men, Ning Xie, Huasheng Li, Bing Yin, Han Li, Lingyun Wang

Language Models

Challenge:	LongLeader aims to assess different LLMs' long-context comprehension abilities . long-constext comprehension is a key bottleneck for many use cases .
Approach:	They propose a leaderboard to assess different LLMs' long-context comprehension abilities . they offer open-source access to the benchmarks and maintain a dedicated website .
Outcome:	The proposed model assesses different LLMs on selected benchmarks and provides open-source access to the benchmarks.

LongAlign: A Recipe for Long Context Alignment of Large Language Models (2024.findings-emnlp)

Copied to clipboard

Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, Juanzi Li

Language Models

Challenge:	Existing studies to build long context language models focus on context extension and continual training on long text.
Approach:	They propose a recipe for instruction fine-tuning on input sequences of similar length . they adopt packing and sorted batching strategies to speed up supervised fine-uning .
Outcome:	The proposed model outperforms existing recipes for LLMs in long context tasks by 30% while maintaining proficiency in handling short, generic tasks.

Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly (2025.coling-main)

Copied to clipboard

Peyman Hosseini, Ignacio Castro, Iacopo Ghinassi, Matthew Purver

Responsible & Trustworthy NLP

Challenge:	Large Language Models (LLMs) have demonstrated remarkable capabilities in comprehending and analyzing lengthy sequential inputs.
Approach:	They propose to implement ad-hoc solutions that enhance LLMs’ performance on long input sequences by up to 50% while reducing API cost and latency by up . to address this limitation, they propose to use three datasets and two tasks to analyze news categorization and sentence analysis to evaluate their models.
Outcome:	The proposed solutions significantly improve LLMs’ performance on long input sequences by up to 50% while reducing API cost and latency by up . to 93% and 50%, respectively.

PEPE: Long-context Extension for Large Language Models via Periodic Extrapolation Positional Encodings (2025.findings-emnlp)

Copied to clipboard

Jikun Hu, Dongsheng Guo, Yuli Liu, Qingyao Ai, Lixuan Wang, Xuebing Sun, Qilei Zhang, Quan Zhou, Cheng Luo

Language Models

Challenge:	Long-context extension attempts to extend contextual window in pre-trained LLMs . primary method involves expanding initial positional encodings, disrupting positional learning .
Approach:	They propose a new extension strategy based on Rotary Position Embedding to extend contextual window in pre-trained large language models.
Outcome:	The proposed method can extend the contextual window in pre-trained large language models . expansion disrupts positional encodings learned during pre-training, authors show .