Relaxing the Constraints: A Dual-Importance Projection Mechanism for Lifelong Model Editing (2026.findings-acl)
Copied to clipboard
Zhenghai Chen, Senbin Xu, Jiaxi Tan, Xinhua Wu, Yan Zhang, Xiawu Zheng, Shengchuan Zhang, Ke Li, Sicheng Zhao, Liujuan Cao, Rongrong Ji
| Challenge: | Existing knowledge editing methods rely on strict orthogonal projection to preserve previously edited knowledge, but this constraint limits gradient expressiveness, resulting in degradation of model generalization and overall performance as the number of edits increases. |
| Approach: | They propose a method that leverages Singular Value Decomposition to identify critical gradient subspaces and introduces a dual mechanism comprising "accumulated importance" and "projection importance" |
| Outcome: | Extensive experiments on five mainstream LLMs show that the proposed method achieves an average comprehensive performance improvement of 10.36% and effectively maintains the model’s general capabilities on downstream tasks. |
Similar Papers
Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for enhancing large language models (LLMs) have achieved some success, but their knowledge understanding and memory capacity significantly degrades after extensive editing. |
| Approach: | They propose a method that stores the basis vectors of the representation space of past edits in a knowledge cache and projects the gradient of the current edit onto a space orthogonal to previous knowledge for updating. |
| Outcome: | The proposed method improves question-answering ability and hallucination mitigation by 14% and 61% for large language models after 3,000 edits. |
HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to lifelong model editing apply parameter perturbations to static and dense layers for all instances. |
| Approach: | They propose a hierarchical reinforcement learning framework that identifies the most knowledge-relevant layers for each editing instance. |
| Outcome: | The proposed framework boosts the performance of the competitive RLEdit by 8.48% with perturbing only half of the layers per edit. |
ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Recent approaches to reduce resource requirements for task-specific large language models have been developed. |
| Approach: | They propose a delta compression approach that optimizes for importance of a model . they use SVD to dynamically adjust the sparsity ratios of different vectors based on their importance . |
| Outcome: | The proposed approach achieves state-of-the-art in retaining task-specific knowledge even at high sparsity ratios. |
GeoEdit: Geometric Knowledge Editing for Large Language Models (2025.emnlp-main)
Copied to clipboard
Yujie Feng, Li-Ming Zhan, Zexin Lu, Yongxin Xu, Xu Chu, Yasha Wang, Jiannong Cao, Philip S. Yu, Xiao-Ming Wu
| Challenge: | Existing training-based model editing methods struggle to incorporate new knowledge while preserving unrelated general knowledge. |
| Approach: | They propose a framework that uses geometric relationships to differentiate between neurons associated with new knowledge updates and those related to general knowledge perturbations. |
| Outcome: | The proposed framework avoids updating neurons with directions approximately orthogonal to existing knowledge, thus preserving the model’s generalization ability. |
Enhancing Translation Ability of Large Language Models by Leveraging Task-Related Layers (2024.lrec-main)
Copied to clipboard
| Challenge: | Experimental validation shows that adjusting task-related layers significantly improves performance on translation tasks while maintaining stability and accuracy on other tasks. |
| Approach: | They propose to adjust task-related layers in large models to better harness their machine translation capabilities by revealing the structure and characteristics of attention weights through singular value decomposition. |
| Outcome: | The proposed method reduces computational resource consumption and catastrophic forgetting while maintaining stability and accuracy on other tasks. |
RuleEdit: Towards Rule-Level Knowledge Generalization to Mitigate Over-Editing in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge editing methods focus on instance-level editing, which is prone to knowledge degradation and general ability deterioration due to redundant instance-specific modifications. |
| Approach: | They propose a rule-level editing method that generalizes rule-derived knowledge to update rule-based instances. |
| Outcome: | The proposed method improves portability and performance over baselines for LLaMA-2-7B on RULEmix. |
Towards Scalable Lifelong Knowledge Editing with Selective Knowledge Suppression (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to modify knowledge are limited due to high training costs and lack stability during sequential edits due to catastrophic forgetting. |
| Approach: | They propose a framework to modify specific knowledge of large language models without retraining the entire model. |
| Outcome: | Extensive experiments on ZSRE, Counterfact, and RIPE show that LightEdit outperforms existing lifelong knowledge editing methods. |
One for All: Update Parameterized Knowledge Across Multiple Models with Once Edit (2025.acl-long)
Copied to clipboard
Weitao Ma, Xiyuan Du, Xiaocheng Feng, Lei Huang, Yichong Huang, Huiyi Zhang, Xiaoliang Yang, Baohang Li, Xiachong Feng, Ting Liu, Bing Qin
| Challenge: | Existing methods for modifying large language models focus on individual models, resulting in errors and hallucinations. |
| Approach: | They propose an ensemble-based approach that employs a plug-in model as the editing module and a dynamic weight mechanism to enhance its effectiveness. |
| Outcome: | The proposed approach outperforms existing methods while achieving superior editing efficiency. |
Forget for Get: A Lightweight Two-phase Gradient Method for Knowledge Editing in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge editing methodologies often encounter parameter conflict during knowledge overwriting and excessive computational overhead. |
| Approach: | They propose a method that erases outdated knowledge and inserts new knowledge at the location that corresponds to the target knowledge. |
| Outcome: | The proposed method achieves more effective knowledge editing at a lower cost compared to previous methods across various base models. |
Resolving UnderEdit & OverEdit with Iterative & Neighbor-Assisted Model Editing (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to update model parameters are limited due to their low efficiency and cost. |
| Approach: | They propose two methods to improve model editing performance by incorporating neighboring knowledge during editing. |
| Outcome: | The proposed methods reduce UnderEdit by 38 percentage points and OverEdit by up to 6 . |