Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Fine-tuning of frontier models can lead to privacy collapse, causing optimisation for helpfulness, exposure to user information, and debugging code printing internal variables. |
| Approach: | They propose to fine-tune frontier models to adapt to specific domains and align with organizational workflows and user preferences. |
| Outcome: | The proposed model fails to perform on safety and utility benchmarks while exhibiting severe privacy vulnerabilities. |
Similar Papers
Unintended Memorization of Sensitive Information in Fine-Tuned Language Models (2026.eacl-long)
Copied to clipboard
Marton Szep, Jorge Marin Ruiz, Georgios Kaissis, Paulina Seidl, Rüdiger von Eisenhart-Rothe, Florian Hinterwimmer, Daniel Rueckert
| Challenge: | Large Language Models (LLMs) on sensitive datasets carry a substantial risk of unintended memorization and leakage of Personally Identifiable Information (PII) prior studies have analyzed memorizing dynamics in LLMs during pre-training and fine-tuning. |
| Approach: | They investigate the vulnerability of PII that appears only in model inputs, not in training targets. |
| Outcome: | The proposed methods show that post-training methods provide more consistent privacy-utility trade-offs . |
The Tug of War Within: Mitigating the Fairness-Privacy Conflicts in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to enhance an LLM's privacy awareness with thousands of samples decrease its fairness awareness. |
| Approach: | They propose a training-free method to Suppress the Privacy and faIrness coupled Neurons (SPIN) which theoretically and empirically decreases the mutual information between fairness and privacy awareness. |
| Outcome: | The proposed method reduces the mutual information between fairness and privacy awareness without compromising general capabilities. |
Large Language Models Can Be Contextual Privacy Protection Learners (2024.emnlp-main)
Copied to clipboard
Yijia Xiao, Yiqiao Jin, Yushi Bai, Yue Wu, Xianjun Yang, Xiao Luo, Wenchao Yu, Xujiang Zhao, Yanchi Liu, Quanquan Gu, Haifeng Chen, Wei Wang, Wei Cheng
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable linguistic comprehension and generation capability, but when applied to specialized industries, they face challenges such as hallucination, insufficient domain knowledge, and failing to incorporate the latest domain knowledge. |
| Approach: | They propose a paradigm for fine-tuning LLMs that effectively injects domain-specific knowledge while safeguarding inference-time data privacy. |
| Outcome: | The proposed model protects private data while enhancing the model's knowledge. |
Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing LLMs require users to submit raw text regardless of its sensitivity, resulting in substantial computational overhead and degrade model performance. |
| Approach: | They propose a new training pipeline that allows a client-side encoder to condition on k-pooled prompt embeddings instead of raw text and a server-side projection module to fine-tune the projection module and LLM on private, domain-specific data using noise-injected embeddables. |
| Outcome: | The proposed approach eliminates the need for transmitting raw prompt text while maintaining a favorable balance between privacy preservation and model utility for both clients and service providers. |
An Empirical Analysis of Memorization in Fine-tuned Autoregressive Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Large language models are shown to present privacy risks through memorization of training data, but little attention has been given to the fine-tuning phase. |
| Approach: | They empirically study memorization of fine-tuning methods using membership inference and extraction attacks and show that fine-timing the head of the model has the highest susceptibility to attacks. |
| Outcome: | The proposed methods have the highest memorization risk, whereas the smaller adapters are less vulnerable to known extraction attacks. |
How to Fine-Tune Safely on a Budget: Model Adaptation Using Minimal Resources (2025.emnlp-industry)
Copied to clipboard
Anh C. Pham, Mihir Thalanki, Michael Sun, Aditya Chaloo, Ankita Gupta, Tian Xia, Aditya Mate, Ehi Nosakhare, Soundararajan Srinivasan
| Challenge: | Existing methods for fine-tuning safety examples are underdeveloped. |
| Approach: | They hypothesize that the effectiveness of a safety example is governed by its instruction-response behavior and its semantic diversity across harm categories. |
| Outcome: | The proposed method reduces harmfulness by up to 41% while adding only 0.05% more data to the fine-tuning set. |
Dissecting Fine-Tuning Unlearning in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning-based unlearning are ineffective at completely erasing model-embedded knowledge, but their true effectiveness remains unclear. |
| Approach: | They propose to use activation patching and parameter restoration experiments to examine the limitations of fine-tuning-based unlearning methods for erasing harmful, sensitive, or copyrighted information within large language models. |
| Outcome: | The proposed methods alter the model’s knowledge retrieval process rather than genuinely erasing the problematic knowledge embedded in the model parameters. |
Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approach (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) enhanced with external contexts face challenges in handling imperfect evidence. |
| Approach: | They propose a framework that can balance internal knowledge with external contexts . they propose gating mechanisms and low-rank representation adapters to adjust hidden representations based on a lightweight intervention function . |
| Outcome: | The proposed model can effectively balance internal knowledge with external context, similar to human cognitive processes. |
Exploring Memorization in Fine-tuned Language Models (2024.acl-long)
Copied to clipboard
Shenglai Zeng, Yaxin Li, Jie Ren, Yiding Liu, Han Xu, Pengfei He, Yue Xing, Shuaiqiang Wang, Jiliang Tang, Dawei Yin
| Challenge: | Existing studies have shown that pre-trained langauge models tend to memorize and regenerate segments of their pre-training corpus when prompted appropriately. |
| Approach: | They conduct the first comprehensive analysis to explore language models’ memorization during fine-tuning across tasks. |
| Outcome: | The proposed analysis shows that memorization presents a strong disparity among different fine-tuning tasks. |
CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuning (2026.acl-long)
Copied to clipboard
| Challenge: | Fine-tuning-as-a-service exposes models to harmful fine-tuneing attacks . however, inherent general adaptability of LLMs allows them to bypass selective unlearning by rapidly relearning or repurposing their general capabilities for harmful tasks. |
| Approach: | They propose a paradigm shift that inducing model collapse instead of selective removal by relearning or repurposing general capabilities for harmful tasks. |
| Outcome: | The proposed model collapse mechanism neutralizes the very general capabilities that attackers exploit, tackling the core issue unaddressed by selective unlearning. |