ULMR: Unlearning Large Language Models via Negative Response and Model Parameter Average (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) have attracted significant interest from the research community due to their broad applicability in many language-oriented tasks. |
| Approach: | They propose a framework which uses pre-training datasets to rewrite instructions and generate negative responses to preserve the performance of the original LLM. |
| Outcome: | The proposed framework can erase the pre-training data while maintaining the performance of the original model. |
Similar Papers
Machine Unlearning of Pre-trained Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Using curated datasets, we establish a robust benchmark for unlearning performance, demonstrating that these methods are over 105 times more computationally efficient than retraining. |
| Approach: | They propose a framework for machine unlearning in pre-trained LLMs and integrate gradient ascent with gradient descent on in-distribution data to achieve robustness. |
| Outcome: | The proposed framework is over 105 times more efficient than retraining on in-distribution data and provides detailed guidelines for efficient hyperparameter tuning in the unlearning process. |
Unlearn What You Want to Forget: Efficient Unlearning for LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be used to memorize a vast amount of data, but can suffer from privacy issues and data protection violations. |
| Approach: | They propose an efficient unlearning framework that could update LLMs without retraining them . they introduce lightweight unlearning layers learned with a selective teacher-student objective into transformers . |
| Outcome: | The proposed framework could update LLMs without having to retrain the whole model after data removals. |
UNLEARN Efficient Removal of Knowledge in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models excel in many tasks but are outperformed by specialized tools for certain tasks. |
| Approach: | They propose a method that uses subspace techniques to selectively remove knowledge . they propose 'unlearn' method that can forget or unlear the knowledge without retraining . |
| Outcome: | The proposed method outperforms existing methods for forgetting target knowledge while preserving related knowledge. |
Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysis (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies have focused on instance-level unlearning, specifically removing predefined instances containing sensitive content. |
| Approach: | They propose a task to erase entity-related knowledge from the target model completely by analyzing the forget set and its size. |
| Outcome: | The proposed task systematically evaluates popular unlearning algorithms and reveals that the knowledge coverage of the forget set and its size play pivotal roles. |
OBLIVIATE: Robust and Practical Machine Unlearning for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) trained over corpora risk memorizing sensitive, copyrighted, or toxic content. |
| Approach: | They propose a framework that removes targeted data while preserving model utility. |
| Outcome: | The proposed framework resists membership inference attacks, minimizes impact on retained data, and maintains robustness across diverse scenarios. |
ReLearn: Unlearning via Learning for Large Language Models (2025.acl-long)
Copied to clipboard
Haoming Xu, Ningyuan Zhao, Liming Yang, Sendong Zhao, Shumin Deng, Mengru Wang, Bryan Hooi, Nay Oo, Huajun Chen, Ningyu Zhang
| Challenge: | Existing methods for unlearning large language models often rely on reverse optimization to reduce target token probabilities. |
| Approach: | They propose a data augmentation and fine-tuning pipeline for effective unlearning . they propose augmentation, evaluation frameworks to measure contextual forgetting . |
| Outcome: | The proposed framework achieves targeted forgetting while preserving high-quality outputs. |
Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models (2025.coling-main)
Copied to clipboard
Anmol Reddy Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid A. Hasan, Elita A.A Lobo
| Challenge: | Existing methods for large language models rely on negative feedback to suppress responses related to the forget set, which often results in nonsensical or inconsistent outputs, diminishing model utility and posing potential privacy risks. |
| Approach: | They propose an approach which combines negative feedback with in-domain positive feedback on the forget set and introduces new evaluation metrics to assess the quality of responses related to the forget sets. |
| Outcome: | The proposed approach avoids undesirable model behaviors while maintaining overall model performance. |
Towards Safer Large Language Models through Machine Unlearning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing work attempted to implement a gradient ascent based approach to prevent LLMs from producing harmful output when faced with problematic prompts. |
| Approach: | They propose a gradient ascent based approach to prevent LLMs from producing harmful output when faced with problematic prompts. |
| Outcome: | The proposed approach eliminates harmful knowledge while preserving utility on normal prompts. |
Harnessing Large Language Models as Post-hoc Correctors (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated their effectiveness in a wide range of tasks, including machine translation and commonsense reasoning. |
| Approach: | They propose a training-free framework that can work as a post-hoc corrector to propose corrections for ML models. |
| Outcome: | The proposed framework improves the performance of a number of models by up to 39% on text analysis and the challenging molecular predictions. |
Soft Prompting for Unlearning in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing ethical and safety considerations for large language models are important for deployment . however, some ethical concerns have been raised due to the presence of private, sensitive, or harmful information in the training data. |
| Approach: | They propose a framework that learns prompt tokens that are prepended to a query to induce unlearning in LLMs. |
| Outcome: | The proposed method improves the trade-off between utility and forgetting for text classification and question-answering. |