Decoupling Memories, Muting Neurons: Towards Practical Machine Unlearning for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for MU degrade model utility, especially when accessing the original training data. |
| Approach: | They propose a method that eliminates the influence of unlearned data by modulating the outputs of merely 1% of the neurons in the feed-forward network modules within the Transformer blocks. |
| Outcome: | The proposed method eliminates the influence of unlearned data from Large Language Models by modulating the outputs of 1% of the neurons in the feed-forward network modules within the Transformer blocks, minimizing disruption to the model’s performance. |
Similar Papers
UNLEARN Efficient Removal of Knowledge in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models excel in many tasks but are outperformed by specialized tools for certain tasks. |
| Approach: | They propose a method that uses subspace techniques to selectively remove knowledge . they propose 'unlearn' method that can forget or unlear the knowledge without retraining . |
| Outcome: | The proposed method outperforms existing methods for forgetting target knowledge while preserving related knowledge. |
Unlearn What You Want to Forget: Efficient Unlearning for LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be used to memorize a vast amount of data, but can suffer from privacy issues and data protection violations. |
| Approach: | They propose an efficient unlearning framework that could update LLMs without retraining them . they introduce lightweight unlearning layers learned with a selective teacher-student objective into transformers . |
| Outcome: | The proposed framework could update LLMs without having to retrain the whole model after data removals. |
Effective Skill Unlearning through Intervention and Abstention (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models exhibit remarkable skills across various domains without training on task-specific datasets. |
| Approach: | They propose two lightweight, training-free machine skill unlearning techniques for LLMs . they propose to unlearning a particular skill while retaining overall capabilities . |
| Outcome: | The proposed methods demonstrate strong unlearning capabilities for the designated skills across seven different languages. |
Machine Unlearning of Pre-trained Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Using curated datasets, we establish a robust benchmark for unlearning performance, demonstrating that these methods are over 105 times more computationally efficient than retraining. |
| Approach: | They propose a framework for machine unlearning in pre-trained LLMs and integrate gradient ascent with gradient descent on in-distribution data to achieve robustness. |
| Outcome: | The proposed framework is over 105 times more efficient than retraining on in-distribution data and provides detailed guidelines for efficient hyperparameter tuning in the unlearning process. |
Towards Robust Evaluation of Unlearning in LLMs via Data Transformations (2024.findings-emnlp)
Copied to clipboard
Abhinav Joshi, Shaswati Saha, Divyaksh Shukla, Sriram Vema, Harsh Jhamtani, Manas Gaur, Ashutosh Modi
| Challenge: | Large Language Models (LLMs) have shown to be a great success in a wide range of applications ranging from regular NLP-based use cases to AI agents. |
| Approach: | They examine the robustness of existing MUL techniques for their ability to enable leakage-proof forgetting in LLMs. |
| Outcome: | The proposed methods can be used to enable leakage-proof forgetting in LLMs. |
Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models and Multimodal Large Language Modells can memorize sensitive information, raising ethical and privacy concerns. |
| Approach: | They propose a novel unlearning framework that selectively clips neurons based on their relative importance to the targeted forget data. |
| Outcome: | The proposed framework selectively clips neurons based on their relative importance to the targeted forget data, curated for different modalities. |
ReLearn: Unlearning via Learning for Large Language Models (2025.acl-long)
Copied to clipboard
Haoming Xu, Ningyuan Zhao, Liming Yang, Sendong Zhao, Shumin Deng, Mengru Wang, Bryan Hooi, Nay Oo, Huajun Chen, Ningyu Zhang
| Challenge: | Existing methods for unlearning large language models often rely on reverse optimization to reduce target token probabilities. |
| Approach: | They propose a data augmentation and fine-tuning pipeline for effective unlearning . they propose augmentation, evaluation frameworks to measure contextual forgetting . |
| Outcome: | The proposed framework achieves targeted forgetting while preserving high-quality outputs. |
Less is KEN: a Universal and Simple Non-Parametric Pruning Algorithm for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing pruning algorithms suffer from limitations such as architecture specificity and reliance on demanding calculations. |
| Approach: | They propose a pruning algorithm based on Kernel Density Estimation . it preserves most significant parameters while restoring others to their pre-training state . |
| Outcome: | The proposed pruning algorithm achieves better performance than the original unpruned version. |
Unlearning Traces the Influential Training Data of Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit remarkable abilities without being explicitly trained for such tasks. |
| Approach: | They propose a method that unlearns a test dataset and evaluates the unlearned model on training datasets. |
| Outcome: | The proposed method resembles UnTrac, while being efficient for massive training datasets. |
Decoding-Unlearning: Fact Forgetting via Entropy-Guided Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for large-scale modeling memorize sensitive information . however, they are limited in real-world scenarios and require updating parameters . |
| Approach: | They propose a training-free, plug-and-play inference-time unlearning strategy that uses a probe to detect queries involving forgettable concepts and applies entropy-guided decoding to suppress target knowledge. |
| Outcome: | Experiments on MUSE, RWKU, and WMDP datasets show that SEGUE outperforms existing methods. |