Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs (2026.acl-long)
Copied to clipboard
Xuyuan Liu, Shengyu Chen, Xinshuai Dong, Yanchi Liu, Xujiang Zhao, Haoyu Wang, Yujun Yan, Haifeng Chen, Zhengzhang Chen
| Challenge: | Large language models (LLMs) produce outdated or inaccurate content. Updating their knowledge efficiently and accurately without costly retraining is a major challenge. |
| Approach: | They propose a robust and scalable method that treats knowledge control as interventions within the model’s representation space. |
| Outcome: | The proposed method achieves fine-grained control over complex, unstructured knowledge while maintaining general utility with frozen base weights. |
Similar Papers
Controllable Memorization in LLMs via Weight Pruning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on mitigating memorization, but the deliberate control of memorisation has been underexplored. |
| Approach: | They propose a gradient-based weight pruning framework to control memorization rates in large language models by fine-grained control over pruning parameters. |
| Outcome: | The proposed framework enables models to suppress or enhance memorization based on application-specific requirements. |
Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for enhancing large language models (LLMs) have achieved some success, but their knowledge understanding and memory capacity significantly degrades after extensive editing. |
| Approach: | They propose a method that stores the basis vectors of the representation space of past edits in a knowledge cache and projects the gradient of the current edit onto a space orthogonal to previous knowledge for updating. |
| Outcome: | The proposed method improves question-answering ability and hallucination mitigation by 14% and 61% for large language models after 3,000 edits. |
Large Language Models with Controllable Working Memory (2023.findings-acl)
Copied to clipboard
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, Sanjiv Kumar
| Challenge: | Large language models (LLMs) have led to a series of breakthroughs in natural language processing due to the massive amounts of world knowledge they memorize during pretraining. |
| Approach: | They propose a method to inject counterfactual and irrelevant contexts into standard supervised datasets to strengthen both controllability and robustness. |
| Outcome: | The proposed method improves controllability and robustness across model architectures and sizes. |
Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Survey (2025.findings-emnlp)
Copied to clipboard
| Challenge: | specialized LLMs are often limited in domain-specific applications that require specialized knowledge. |
| Approach: | They provide a comprehensive overview of four key methods to enhance large language models by integrating domain-specific knowledge. |
| Outcome: | The proposed methods are categorized into four key approaches: dynamic knowledge injection, static knowledge embedding, modular adapters, and prompt optimization. |
Continual Learning of Large Language Models (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial explores the challenges of continual learning in large language models . participants will learn strategies to mitigate forgetting and manage data and evaluation pipelines . |
| Approach: | This tutorial offers a comprehensive exploration of continual learning in the context of large language models. |
| Outcome: | This tutorial explores the challenges of continual learning in large language models . participants will learn how to manage data and evaluation pipelines and adapt responsibly . |
Knowledge Graph-Driven Memory Editing with Directional Interventions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are hampered by inaccuracies and outdated information. |
| Approach: | They propose a framework that constructs knowledge graphs using available information to guide the direction of knowledge editing. |
| Outcome: | The proposed framework allows consistent, aligned, and stable information during large-scale editing scenarios. |
RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging (2025.emnlp-main)
Copied to clipboard
Bowen Wang, Haiyuan Wan, Liwen Shi, Chen Yang, Peng He, Yue Ma, Haochen Han, Wenhao Li, Tiao Tan, Yongjian Li, Fangming Liu, Gong Yifan, Sheng Zhang
| Challenge: | Existing models that require task labels or performance trade-offs are susceptible to catastrophic forgetting. |
| Approach: | They propose a representation-aware model merging framework for continual learning without access to historical data. |
| Outcome: | The proposed framework outperforms baselines in knowledge retention and generalization across five NLP tasks and multiple continual learning scenarios. |
InfiniPot: Infinite Context Processing on Memory-Constrained LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | InfiniPot is a KV cache control framework that can handle long input contexts without additional training. |
| Approach: | They propose a KV cache control framework that can handle long input contexts efficiently without additional training. |
| Outcome: | The proposed framework outperforms models trained for long contexts in various NLP tasks and is highly efficient and versatile. |
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are impressive in solving tasks, but they can quickly be outdated after deployment. |
| Approach: | They provide a review of recent advances in aligning deployed large language models with the ever-changing world knowledge. |
| Outcome: | The proposed models can be used to perform various tasks directly through in-context learning or for further fine-tuning for domain-specific uses. |
Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approach (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) enhanced with external contexts face challenges in handling imperfect evidence. |
| Approach: | They propose a framework that can balance internal knowledge with external contexts . they propose gating mechanisms and low-rank representation adapters to adjust hidden representations based on a lightweight intervention function . |
| Outcome: | The proposed model can effectively balance internal knowledge with external context, similar to human cognitive processes. |