| Challenge: | Existing selective methods that focus on identifying token-level or span-level unlearning targets are misaligning unlearning objectives with the model’s internal behavior. |
| Approach: | They propose a selective method that uses model-intrinsic information to identify token-level or span-level unlearning targets within a text rather than entire sequences. |
| Outcome: | The proposed method achieves comparable unlearning performance while significantly better preserving retained knowledge. |
Similar Papers
Not Every Token Needs Forgetting: Selective Unlearning Balancing Forgetting and Utility in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Conventional unlearning approaches forget all tokens in a target document, including common tokens that carry general knowledge. |
| Approach: | They propose a method that identifies a critical subset of tokens within the forgetting set that is relevant to the unwanted information and unlearns only those tokens. |
| Outcome: | Experiments on two benchmarks and six baseline unlearning algorithms show that selective unlearning achieves effective unlearning on the targeted forget data. |
Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysis (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies have focused on instance-level unlearning, specifically removing predefined instances containing sensitive content. |
| Approach: | They propose a task to erase entity-related knowledge from the target model completely by analyzing the forget set and its size. |
| Outcome: | The proposed task systematically evaluates popular unlearning algorithms and reveals that the knowledge coverage of the forget set and its size play pivotal roles. |
BLUR: A Bi-Level Optimization Approach for LLM Unlearning (2026.eacl-long)
Copied to clipboard
Hadi Reisizadeh, Jinghan Jia, Zhiqi Bu, Bhanukiran Vinzamuri, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, Sijia Liu, Mingyi Hong
| Challenge: | Existing algorithms to unlearn knowledge and capabilities from large datasets are unclear how to best formulate the unlearning problem. |
| Approach: | They propose to model the hierarchical structure of the unlearning problem, where the forget problem takes priority over the retain problem, and propose an algorithm that aims to unlearn knowledge and capabilities. |
| Outcome: | The proposed algorithm outperforms all state-of-the-art algorithms across unlearning tasks, models, and metrics. |
Direct Token Optimization: A Self-Contained Approach to Large Language Model Unlearning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for large language models (LLMs) rely on external resources such as auxiliary models, retain datasets, or even commercial AI services. |
| Approach: | They propose a self-contained unlearning approach that optimizes the token-level objectives to unlearn specific sequences without external resources. |
| Outcome: | The proposed approach improves the forget quality up to 16.8 over the latest benchmarks while maintaining comparable model utility. |
Hit the Sweet Spot! Span-Level Ensemble for Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | a recent study focused on sample-level and token-level ensembles, which hinder dynamic correction and enhancement of outputs during the generation process. |
| Approach: | They propose a span-level ensemble method that balances real-time adjustments and accurate ensemble decisions. |
| Outcome: | The proposed method improves performance across language generation tasks significantly. |
Improving Span Representation by Efficient Span-Level Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating high-quality span representations are limited by subset of tokens . span-span interactions should play an important role in span encoding, authors argue . |
| Approach: | They propose to introduce span-span interactions and more comprehensive span-token interactions to improve span representations. |
| Outcome: | The proposed model outperforms baseline models on span-related tasks and shows superior performance. |
Forget What Matters, Keep the Rest: Selective Unlearning of Informative Tokens (2026.acl-long)
Copied to clipboard
| Challenge: | Recent studies have explored token-wise loss regularizers that prioritize informative tokens, but rely on ground-truth confidence or external linguistic parsers, which limits their ability to capture contextual information or the model’s overall predictive state. |
| Approach: | They propose an Entropy-guided Token Weighting (ETW) token-level unlearning regularizer that uses entropy of the predictive distribution as a proxy for token informativeness. |
| Outcome: | The proposed token-level unlearning regularizer can achieve more effective unlearning while better preserving model utility than existing token-based approaches. |
UNLEARN Efficient Removal of Knowledge in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models excel in many tasks but are outperformed by specialized tools for certain tasks. |
| Approach: | They propose a method that uses subspace techniques to selectively remove knowledge . they propose 'unlearn' method that can forget or unlear the knowledge without retraining . |
| Outcome: | The proposed method outperforms existing methods for forgetting target knowledge while preserving related knowledge. |
Self-training Large Language Models through Knowledge Detection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) often require extensive labeled datasets and training compute to achieve impressive performance across downstream tasks. |
| Approach: | They propose a self-training paradigm where the LLM curates its own labels and selectively trains on unknown data samples identified through a reference-free consistency method. |
| Outcome: | The proposed model reduces the dependency on large labeled datasets and mitigates catastrophic forgetting in out-of-distribution benchmarks. |
Answer When Needed, Forget When Not: Language Models Pretend to Forget via In-Context Knowledge Unlearning (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly required to selectively unlearn specific information. |
| Approach: | They propose a method which fine-tunes pre-trained LLMs to enable prompt unlearning of target knowledge within the context while preserving unrelated information. |
| Outcome: | The proposed method achieves up to 95% forget accuracy while retaining 80% of unrelated knowledge, significantly outperforming baselines in both in-domain and out-of-domain scenarios. |