InfiniteICL: Breaking the Limit of Context Window Size via Long Short-term Memory Transformation (2025.findings-acl)
Copied to clipboard
| Challenge: | InfiniteICL is a framework that parallels context and parameters in large language models with short- and long-term memory in human cognitive systems. |
| Approach: | They propose a framework that parallels context and parameters in large language models with short- and long-term memory in human cognitive systems and enables infinite context integration. |
| Outcome: | The proposed framework reduces context length by 90% while achieving 103% average performance of full-context prompting across fact recall, grounded reasoning, and skill acquisition tasks. |
Similar Papers
LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Currently, large language models (LLMs) train on short text segments due to the computational overhead quadratic in the input lengths of their Transformer architectures. |
| Approach: | They propose a method that allows LLMs pre-trained with 2K or 4K-long segments to generalize to up to 200M length inputs while retaining perplexity. |
| Outcome: | The proposed method achieves 2.7 decoding speed up and 7.5 memory saving over the original model. |
In-context Continual Learning Assisted by an External Continual Learner (2025.coling-main)
Copied to clipboard
| Challenge: | Existing continual learning methods suffer from catastrophic forgetting (CF) . Existing methods rely on fine-tuning or adapting large language models (LLMs) |
| Approach: | They propose an approach that integrates an external continual learner (ECL) with ICL to enable scalable CL without catastrophic forgetting (CF). |
| Outcome: | The proposed approach outperforms existing baselines while maintaining high performance. |
Rethinking Long Context Generation from the Continual Learning Perspective (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) struggle with processing long contexts due to the limited context window. |
| Approach: | They propose to combine a limited context window with a continual learning perspective to improve LLMs' efficiency in processing long contexts. |
| Outcome: | The proposed models improve the performance of Large Language Models (LLMs) by integrating learning strategies with existing approaches. |
OpenICL: An Open-Source Framework for In-context Learning (2023.acl-demo)
Copied to clipboard
| Challenge: | In-context Learning (ICL) is a new paradigm for large language model evaluation. |
| Approach: | They propose an open-source toolkit for ICL and LLM evaluation. |
| Outcome: | The proposed framework is highly flexible and flexible and can be easily combined with other tools to suit users' needs. |
Revisiting In-Context Learning with Long Context Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | In-Context Learning (ICL) is a technique by which language models make predictions based on examples provided in their input context. |
| Approach: | They revisited previous studies using in-context learning techniques . they found that using a data augmentation approach, they significantly improved ICL performance . |
| Outcome: | The proposed approach significantly improves ICL performance on 18 datasets spanning 4 tasks . the proposed approach does not improve performance over a simple random sample selection method . |
InfiniPot: Infinite Context Processing on Memory-Constrained LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | InfiniPot is a KV cache control framework that can handle long input contexts without additional training. |
| Approach: | They propose a KV cache control framework that can handle long input contexts efficiently without additional training. |
| Outcome: | The proposed framework outperforms models trained for long contexts in various NLP tasks and is highly efficient and versatile. |
Parallel Context Windows for Large Language Models (2023.acl-long)
Copied to clipboard
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham
| Challenge: | Existing efforts to address context window limitation for off-the-shelf LLMs involve training specialized architectures. |
| Approach: | They propose a method that carves a long context into chunks and restricts attention to apply only within each window. |
| Outcome: | The proposed method shows significant improvements on in-context learning tasks with diverse input and output spaces. |
In-Context Learning with Long-Context Models: An In-Depth Exploration (2025.naacl-long)
Copied to clipboard
| Challenge: | In-context learning is limited by context length, but it can be used for many tasks. |
| Approach: | They study the behavior of in-context learning at an extreme context length . example retrieval shows excellent performance at low context lengths but has diminished gains . |
| Outcome: | The proposed model can perform many tasks with reasonable accuracy when a few examples are provided in-context. |
Naive Bayes-based Context Extension for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown promising in-context learning abilities, but are often impeded by length limitations of transformer architecture. |
| Approach: | They propose a framework to enable existing Large Language Models to perform ICL with an increased number of demonstration examples by significantly expanding their context size. |
| Outcome: | The proposed framework outperforms existing methods in in-context learning with an increased number of demonstration examples while preserving linear efficiency. |
LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models (2025.acl-long)
Copied to clipboard
Zhiyuan Hu, Yuliang Liu, Jinman Zhao, Suyuchen Wang, WangYan WangYan, Wei Shen, Qing Gu, Anh Tuan Luu, See-Kiong Ng, Zhiwei Jiang, Bryan Hooi
| Challenge: | Large language models face significant challenges in handling long-context tasks because of their limited effective context window size during pretraining, which restricts their ability to generalize over extended sequences. |
| Approach: | They propose a training strategy for extending the context window of LLMs including impactful token analysis, position index transformation, and training optimization strategies. |
| Outcome: | Experiments on three types of LLMs show that LongRecipe can utilize long sequences while requiring only 30% of the target context window size. |