Ethos: Rectifying Language Models in Orthogonal Parameter Space (2024.findings-naacl)
Copied to clipboard
| Challenge: | Language models (LMs) generate toxic, biased content and reveal private training records. |
| Approach: | They propose an efficient approach that rectifies LMs to mitigate toxicity and bias . Ethos distinguishes general beneficial and undesired knowledge when reconstructing task vectors . |
| Outcome: | The proposed approach mitigates toxicity and bias in outputs and avoids privacy leakage. |
Similar Papers
Bias Vector: Mitigating Biases in Language Models with Task Arithmetic Approach (2025.coling-main)
Copied to clipboard
| Challenge: | Using language models (LMs) has increased in use, and the use of biases and stereotypes is creating social problems. |
| Approach: | They propose a method to mitigate LM biases by continual training on biased data . they use masked language modeling to construct a Bias Vector as the difference between biased LMs and pre-trained LM weights . |
| Outcome: | The proposed method improves on the GLUE and SEAT benchmarks. |
ULMR: Unlearning Large Language Models via Negative Response and Model Parameter Average (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) have attracted significant interest from the research community due to their broad applicability in many language-oriented tasks. |
| Approach: | They propose a framework which uses pre-training datasets to rewrite instructions and generate negative responses to preserve the performance of the original LLM. |
| Outcome: | The proposed framework can erase the pre-training data while maintaining the performance of the original model. |
Intrinsic Test of Unlearning Using Parametric Knowledge Traces (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for “unlearning” information captured in large language models rely on behavioral tests without monitoring residual knowledge in model parameters. |
| Approach: | They propose a general evaluation methodology that uses vocabulary projections to inspect concepts encoded in model parameters. |
| Outcome: | The proposed method detects changes in parametric traces of unlearned concepts and localizes them in two open-source LLMs. |
REVS: Unlearning Sensitive Information in Language Models via Rank Editing in the Vocabulary Space (2025.findings-acl)
Copied to clipboard
| Challenge: | Current approaches to address this issue involve costly dataset scrubbing or model filtering through unlearning and model editing. |
| Approach: | They propose a method for unlearning sensitive information from language models . they curate email and URL datasets and a social security number dataset . |
| Outcome: | The proposed method shows superior performance and robustness to extraction attacks on real-world datasets. |
Forget What Matters, Keep the Rest: Selective Unlearning of Informative Tokens (2026.acl-long)
Copied to clipboard
| Challenge: | Recent studies have explored token-wise loss regularizers that prioritize informative tokens, but rely on ground-truth confidence or external linguistic parsers, which limits their ability to capture contextual information or the model’s overall predictive state. |
| Approach: | They propose an Entropy-guided Token Weighting (ETW) token-level unlearning regularizer that uses entropy of the predictive distribution as a proxy for token informativeness. |
| Outcome: | The proposed token-level unlearning regularizer can achieve more effective unlearning while better preserving model utility than existing token-based approaches. |
Language Models Implement Simple Word2Vec-style Vector Arithmetic (2024.naacl-long)
Copied to clipboard
| Challenge: | a primary criticism of language models is their inscrutability. |
| Approach: | They propose to use a vector arithmetic style mechanism to solve relational tasks . they find that this mechanism is specific to tasks that require retrieval from pretraining memory . |
| Outcome: | The proposed model reduces to a simple additive update for a variety of tasks . the findings contribute to proving that the models are interpretable and reliable . |
OBLIVIATE: Robust and Practical Machine Unlearning for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) trained over corpora risk memorizing sensitive, copyrighted, or toxic content. |
| Approach: | They propose a framework that removes targeted data while preserving model utility. |
| Outcome: | The proposed framework resists membership inference attacks, minimizes impact on retained data, and maintains robustness across diverse scenarios. |
Extracting Latent Steering Vectors from Pretrained Language Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Prior work on controllable text generation has focused on learning how to control language models through trainable decoding, smart-prompt design, or fine-tuning based on a desired objective. |
| Approach: | They propose to extract latent vectors directly from pretrained language model decoders without fine-tuning. |
| Outcome: | The proposed approach generates a target sentence nearly perfectly for English sentences . it outperforms pooled hidden states of models on a textual similarity benchmark . |
Knowledge Unlearning for Mitigating Privacy Risks in Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work shows that an adversary can extract training data from Pretrained Language Models including Personally Identifiable Information (PII) such as names, phone numbers, and email addresses. |
| Approach: | They propose to use knowledge unlearning to reduce privacy risks for LMs by performing gradient ascent on target token sequences instead of trying to unlearn all the data at once. |
| Outcome: | The proposed method can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust. |
Towards Safer Large Language Models through Machine Unlearning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing work attempted to implement a gradient ascent based approach to prevent LLMs from producing harmful output when faced with problematic prompts. |
| Approach: | They propose a gradient ascent based approach to prevent LLMs from producing harmful output when faced with problematic prompts. |
| Outcome: | The proposed approach eliminates harmful knowledge while preserving utility on normal prompts. |