LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Long-context modeling has drawn more attention in the area of Large Language Models (LLMs). |
| Approach: | They propose a Long-context data selection framework with Attention-based Dependency Measurement which can efficiently identify high-quality long-contrast data from a large-scale, multi-domain pre-training corpus. |
| Outcome: | The proposed framework significantly boosts the performance of LLMs on multiple long-context tasks with only 1B tokens for continual training. |
Similar Papers
LongAttn: Selecting Long-context Training Data via Token-level Attention (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to select long-context data often rely on sentence-level analysis, which can be greatly optimized in both performance and efficiency. |
| Approach: | They propose a token-level framework which quantifies long-range dependencies for LLMs by calculating token-based dependency strength and distribution uniformity of token scores. |
| Outcome: | The proposed framework quantifies long-range dependencies, enabling more accurate and efficient data selection. |
Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Long-context modeling capabilities are important for large language models (LLMs) however, training LLMs with long context windows is insufficient since some samples do not exhibit strong semantic dependencies across long contexts. |
| Approach: | They propose a data mining framework ProLong that assigns each training sample with a long dependency score and ranks and filters them according to their results. |
| Outcome: | The proposed framework can rank and filter training samples that exhibit more powerful long-context modeling abilities. |
Effective Long-Context Scaling of Foundation Models (2024.naacl-long)
Copied to clipboard
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, Hao Ma
| Challenge: | Large language models (LLMs) are rapidly deployed and continue to evolve through scaling. |
| Approach: | They propose a method to train strong long-context LLMs that are capable of utilizing massive context windows of up to 32,000 tokens. |
| Outcome: | The proposed model can surpass gpt-3.5-turbo-16k's overall performance on long-context benchmarks with a cost-effective instruction tuning procedure that is free of expensive annotations. |
How to Train Long-Context Language Models (Effectively) (2025.acl-long)
Copied to clipboard
| Challenge: | a new study shows that language models can process extremely long contexts with minimal training. |
| Approach: | They use supervised fine-tuning and continued training to evaluate a language model's long-context capabilities. |
| Outcome: | The proposed model outperforms Llama-3.1-8B-Instruct on most long-context tasks . the model can process 512K tokens, one of the longest context windows of LMs . |
Rethinking Long Context Generation from the Continual Learning Perspective (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) struggle with processing long contexts due to the limited context window. |
| Approach: | They propose to combine a limited context window with a continual learning perspective to improve LLMs' efficiency in processing long contexts. |
| Outcome: | The proposed models improve the performance of Large Language Models (LLMs) by integrating learning strategies with existing approaches. |
LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models (2025.acl-long)
Copied to clipboard
Zhiyuan Hu, Yuliang Liu, Jinman Zhao, Suyuchen Wang, WangYan WangYan, Wei Shen, Qing Gu, Anh Tuan Luu, See-Kiong Ng, Zhiwei Jiang, Bryan Hooi
| Challenge: | Large language models face significant challenges in handling long-context tasks because of their limited effective context window size during pretraining, which restricts their ability to generalize over extended sequences. |
| Approach: | They propose a training strategy for extending the context window of LLMs including impactful token analysis, position index transformation, and training optimization strategies. |
| Outcome: | Experiments on three types of LLMs show that LongRecipe can utilize long sequences while requiring only 30% of the target context window size. |
CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search (2025.naacl-industry)
Copied to clipboard
Kaixin Wu, Yixin Ji, Zeyuan Chen, Qiang Wang, Cunxiang Wang, Hong Liu, Baijun Ji, Xu Jia, Zhongyi Liu, Jinjie Gu, Yuan Zhou, Linjian Mo
| Challenge: | Relevance modeling between queries and items is a key component of commercial search engines. |
| Approach: | They propose a framework for continual pre-training of LLMs to enhance domain knowledge . they employ queries and multi-field item to jointly pre-train for enhancing domain knowledge. |
| Outcome: | The proposed model achieves convincing performance compared to strong baselines. |
Re3Syn: A Dependency-Based Data Synthesis Framework for Long-Context Post-training (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for constructing long-context data by concatenating short documents have overlooked a crucial characteristic of long-constituency data quality, semantic dependency. |
| Approach: | They propose a framework called Retrieval, Dependency Recognition, and Reorder for data synthesis which leverages semantic similarity to retrieve relevant documents and form several batches. |
| Outcome: | The proposed framework leverages semantic similarity to retrieve relevant documents and form several batches. |
EpMAN: Episodic Memory AttentioN for Generalizing to Longer Contexts (2025.acl-long)
Copied to clipboard
Subhajit Chaudhury, Payel Das, Sarathkrishna Swaminathan, Georgios Kollias, Elliot Nelson, Khushbu Pahwa, Tejaswini Pedapati, Igor Melnyk, Matthew Riemer
| Challenge: | Recent advances in Large Language Models (LLMs) have yielded impressive successes on many language tasks, but efficient processing of long contexts remains a significant challenge. |
| Approach: | They propose a method for processing long contexts in an episodic memory module while holistically attending to semantically-relevant context chunks. |
| Outcome: | The proposed method outperforms baseline decoders on multiple long-context recall and question-answering benchmarks on 16k to 256k tokens. |
LongHeads: Multi-Head Attention is Secretly a Long Context Processor (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models struggle to process lengthy inputs due to limited length generalization and attention’s quadratic computational demands. |
| Approach: | They propose a training-free framework that allows each head to attend to important context chunks instead of allowing each head a full sentence . |
| Outcome: | The proposed framework unlocks multi-head attention's untapped potential by allowing each head to attend to important context chunks instead of the full sentence. |