Forget for Get: A Lightweight Two-phase Gradient Method for Knowledge Editing in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge editing methodologies often encounter parameter conflict during knowledge overwriting and excessive computational overhead. |
| Approach: | They propose a method that erases outdated knowledge and inserts new knowledge at the location that corresponds to the target knowledge. |
| Outcome: | The proposed method achieves more effective knowledge editing at a lower cost compared to previous methods across various base models. |
Similar Papers
CoME: An Unlearning-based Approach to Conflict-free Model Editing (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) often retain outdated or incorrect information from pre-training, which undermines their reliability. |
| Approach: | They propose a conflict-free model editing framework that selectively removes outdated knowledge from LLMs to improve their accuracy and reliability. |
| Outcome: | The proposed framework improves both editing accuracy and model reliability when applied to existing editing methods. |
Knowledge Graph-Driven Memory Editing with Directional Interventions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are hampered by inaccuracies and outdated information. |
| Approach: | They propose a framework that constructs knowledge graphs using available information to guide the direction of knowledge editing. |
| Outcome: | The proposed framework allows consistent, aligned, and stable information during large-scale editing scenarios. |
Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for enhancing large language models (LLMs) have achieved some success, but their knowledge understanding and memory capacity significantly degrades after extensive editing. |
| Approach: | They propose a method that stores the basis vectors of the representation space of past edits in a knowledge cache and projects the gradient of the current edit onto a space orthogonal to previous knowledge for updating. |
| Outcome: | The proposed method improves question-answering ability and hallucination mitigation by 14% and 61% for large language models after 3,000 edits. |
Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown impressive capabilities in various downstream tasks but typically face Catastrophic Forgetting (CF) during fine-tuning. |
| Approach: | They propose a pruning-based approach to balance CF and downstream task performance by integrating the ratio of the task vector to pre-trained model parameters into the pruning criteria. |
| Outcome: | The proposed pruning-based approach limits CF to just 0.25% while maintaining 99.67% accuracy on downstream tasks. |
UNLEARN Efficient Removal of Knowledge in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models excel in many tasks but are outperformed by specialized tools for certain tasks. |
| Approach: | They propose a method that uses subspace techniques to selectively remove knowledge . they propose 'unlearn' method that can forget or unlear the knowledge without retraining . |
| Outcome: | The proposed method outperforms existing methods for forgetting target knowledge while preserving related knowledge. |
Unlearn What You Want to Forget: Efficient Unlearning for LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be used to memorize a vast amount of data, but can suffer from privacy issues and data protection violations. |
| Approach: | They propose an efficient unlearning framework that could update LLMs without retraining them . they introduce lightweight unlearning layers learned with a selective teacher-student objective into transformers . |
| Outcome: | The proposed framework could update LLMs without having to retrain the whole model after data removals. |
LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin (2024.acl-long)
Copied to clipboard
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Experimental results show that, as the instruction data increases, LoRAMoE can significantly improve the ability to process downstream tasks, while maintaining the world knowledge stored in the LLM. |
| Approach: | They propose a framework that introduces several low-rank adapters and integrates them by using a router network to freeze the backbone model and force a portion of LoRAs to focus on leveraging world knowledge to solve downstream tasks. |
| Outcome: | The proposed framework freezes the backbone model and forces a portion of LoRAs to focus on leveraging world knowledge to solve downstream tasks, to alleviate world knowledge forgetting. |
Answer When Needed, Forget When Not: Language Models Pretend to Forget via In-Context Knowledge Unlearning (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly required to selectively unlearn specific information. |
| Approach: | They propose a method which fine-tunes pre-trained LLMs to enable prompt unlearning of target knowledge within the context while preserving unrelated information. |
| Outcome: | The proposed method achieves up to 95% forget accuracy while retaining 80% of unrelated knowledge, significantly outperforming baselines in both in-domain and out-of-domain scenarios. |
Knowledge Editing for Large Language Models (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are not immune to issues of factual accuracy or logically consistent. |
| Approach: | This tutorial will present cutting-edge methods and practical tools for editing Large Language Models (LLMs). |
| Outcome: | The aim of this course is to familiarize researchers with the latest advancements and emerging strategies in the realm of knowledge editing for LLMs. |
Learning to Edit: Aligning LLMs with Knowledge Editing (2024.acl-long)
Copied to clipboard
Yuxin Jiang, Yufei Wang, Chuhan Wu, Wanjun Zhong, Xingshan Zeng, Jiahui Gao, Liangyou Li, Xin Jiang, Lifeng Shang, Ruiming Tang, Qun Liu, Wei Wang
| Challenge: | Existing knowledge editing techniques rely on memorizing updated knowledge, impeding LLMs from effectively combining the new knowledge with their inherent knowledge when answering questions. |
| Approach: | They propose a Learning to Edit framework that equips LLMs with the ability to apply updated knowledge to input questions through a two-phase process . |
| Outcome: | The proposed framework outperforms existing methods in knowledge editing tasks and compares it with four benchmarks and two LLM architectures. |