Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for unlearning in large language models often hallucinate, generate abnormal token sequences, or behave inconsistently, raising safety and trust concerns. |
| Approach: | They propose a formal definition of unlearning honesty that preserves both utility and honesty on retained knowledge and ensures effective forgetting while encouraging the model to acknowledge its limitations. |
| Outcome: | The proposed method achieves highest rejection rate and refusal stability on Q A tasks from the forget set, nearly double the second-best method. |
Similar Papers
Parametric Knowledge is Not All You Need: Toward Honest Large Language Models via Retrieval of Pretraining Data (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models are highly capable of answering questions, but they are often unaware of their own knowledge boundary, i.e., knowing what they know and what they don’t know. |
| Approach: | They propose a method to evaluate LLM honesty using Pythia with publicly available pretraining data. |
| Outcome: | The proposed method is based on Pythia, a truly open LLM with publicly available pretraining data. |
Reveal and Release: Iterative LLM Unlearning with Self-generated Data (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to unlearning large language models assume full access to the forget dataset, overlooking two key challenges: (1) Forget data is often privacy-sensitive, rare, or legally regulated, making it expensive or impractical to obtain (2) The distribution of available forget data may not align with how that information is represented within the model. |
| Approach: | They propose a “Reveal-and-Release” method to unlearn with self-generated data, prompting the model to reveal what it knows using optimized instructions. |
| Outcome: | The proposed method removes the influence of undesirable data from the model. |
BLUR: A Bi-Level Optimization Approach for LLM Unlearning (2026.eacl-long)
Copied to clipboard
Hadi Reisizadeh, Jinghan Jia, Zhiqi Bu, Bhanukiran Vinzamuri, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, Sijia Liu, Mingyi Hong
| Challenge: | Existing algorithms to unlearn knowledge and capabilities from large datasets are unclear how to best formulate the unlearning problem. |
| Approach: | They propose to model the hierarchical structure of the unlearning problem, where the forget problem takes priority over the retain problem, and propose an algorithm that aims to unlearn knowledge and capabilities. |
| Outcome: | The proposed algorithm outperforms all state-of-the-art algorithms across unlearning tasks, models, and metrics. |
Unlearn What You Want to Forget: Efficient Unlearning for LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be used to memorize a vast amount of data, but can suffer from privacy issues and data protection violations. |
| Approach: | They propose an efficient unlearning framework that could update LLMs without retraining them . they introduce lightweight unlearning layers learned with a selective teacher-student objective into transformers . |
| Outcome: | The proposed framework could update LLMs without having to retrain the whole model after data removals. |
Answer When Needed, Forget When Not: Language Models Pretend to Forget via In-Context Knowledge Unlearning (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly required to selectively unlearn specific information. |
| Approach: | They propose a method which fine-tunes pre-trained LLMs to enable prompt unlearning of target knowledge within the context while preserving unrelated information. |
| Outcome: | The proposed method achieves up to 95% forget accuracy while retaining 80% of unrelated knowledge, significantly outperforming baselines in both in-domain and out-of-domain scenarios. |
Alleviating Hallucinations from Knowledge Misalignment in Large Language Models via Selective Abstention Learning (2025.acl-long)
Copied to clipboard
Lei Huang, Xiaocheng Feng, Weitao Ma, Yuchun Fan, Xiachong Feng, Yuxuan Gu, Yangfan Ye, Liang Zhao, Weihong Zhong, Baoxin Wang, Dayong Wu, Guoping Hu, Lingpeng Kong, Tong Xiao, Ting Liu, Bing Qin
| Challenge: | Large language models (LLMs) suffer from severe hallucination issues due to the knowledge misalignment between the pre-training stage and the supervised fine-tuning stage. |
| Approach: | They propose a training objective with an abstention mechanism that selectively rejects tokens that misalign with the desired knowledge distribution via a special [REJ] token. |
| Outcome: | The proposed model selectively rejects tokens that misalign with the desired knowledge distribution via a special [REJ] token. |
Not Every Token Needs Forgetting: Selective Unlearning Balancing Forgetting and Utility in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Conventional unlearning approaches forget all tokens in a target document, including common tokens that carry general knowledge. |
| Approach: | They propose a method that identifies a critical subset of tokens within the forgetting set that is relevant to the unwanted information and unlearns only those tokens. |
| Outcome: | Experiments on two benchmarks and six baseline unlearning algorithms show that selective unlearning achieves effective unlearning on the targeted forget data. |
Machine Unlearning of Pre-trained Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Using curated datasets, we establish a robust benchmark for unlearning performance, demonstrating that these methods are over 105 times more computationally efficient than retraining. |
| Approach: | They propose a framework for machine unlearning in pre-trained LLMs and integrate gradient ascent with gradient descent on in-distribution data to achieve robustness. |
| Outcome: | The proposed framework is over 105 times more efficient than retraining on in-distribution data and provides detailed guidelines for efficient hyperparameter tuning in the unlearning process. |
Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models (2025.coling-main)
Copied to clipboard
Anmol Reddy Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid A. Hasan, Elita A.A Lobo
| Challenge: | Existing methods for large language models rely on negative feedback to suppress responses related to the forget set, which often results in nonsensical or inconsistent outputs, diminishing model utility and posing potential privacy risks. |
| Approach: | They propose an approach which combines negative feedback with in-domain positive feedback on the forget set and introduces new evaluation metrics to assess the quality of responses related to the forget sets. |
| Outcome: | The proposed approach avoids undesirable model behaviors while maintaining overall model performance. |
Learn and Unlearn: Addressing Misinformation in Multilingual LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to unlearning large language models (LLMs) focus on English data, but they ignore multilingual contexts and can produce misleading, offensive, or otherwise fake content. |
| Approach: | They investigate the propagation of information in multilingual large language models and evaluate unlearning methods to address harmful content in multi-lingual contexts. |
| Outcome: | The proposed methods can effectively eliminate harmful content for all languages by addressing both English and the original language of the harmful data. |