Estimating Privacy Leakage of Augmented Contextual Knowledge in Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Prior work has shown that privacy leakage of parametric knowledge often occurs from memorized pre-training data. |
| Approach: | They propose a metric that builds on differential privacy to estimate the privacy leakage of contextual knowledge during decoding by comparing parametric and contextual knowledge. |
| Outcome: | The proposed method overestimates the privacy leakage of parametric knowledge while separating parametric and contextual knowledge. |
Similar Papers
Adaptive Backtracking for Privacy Protection in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing privacy protection methods are prone to privacy leakage, but they are not effective in ensuring the privacy of users. |
| Approach: | They propose to capture latent leakage tendency of large language models during generation process and to construct a new benchmark for personal information. |
| Outcome: | The proposed method improves privacy by up to 14% over strong baselines against adversarial attacks, avoiding the degradation of response utility. |
Large Language Models Can Be Contextual Privacy Protection Learners (2024.emnlp-main)
Copied to clipboard
Yijia Xiao, Yiqiao Jin, Yushi Bai, Yue Wu, Xianjun Yang, Xiao Luo, Wenchao Yu, Xujiang Zhao, Yanchi Liu, Quanquan Gu, Haifeng Chen, Wei Wang, Wei Cheng
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable linguistic comprehension and generation capability, but when applied to specialized industries, they face challenges such as hallucination, insufficient domain knowledge, and failing to incorporate the latest domain knowledge. |
| Approach: | They propose a paradigm for fine-tuning LLMs that effectively injects domain-specific knowledge while safeguarding inference-time data privacy. |
| Outcome: | The proposed model protects private data while enhancing the model's knowledge. |
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prior attempts at measuring leakage of MLMs via membership inference attacks have been inconclusive, implying potential robustness of Mlms to privacy attacks. |
| Approach: | They propose a stronger membership inference attack based on likelihood ratio hypothesis testing that involves an additional MLM to more accurately quantify the privacy risks of memorization in MLMs. |
| Outcome: | The proposed attack improves the AUC of prior membership inference attacks from 0.66 to an alarmingly high 0.90 level on models trained on medical notes. |
How Training Data Shapes the Use of Parametric and In-Context Knowledge in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models leverage parametric and in-context knowledge in training . however, when these sources conflict, models arbitrate based on their internal confidence . |
| Approach: | They conduct controlled experiments using synthetic corpora to identify data properties that shape knowledge utilization. |
| Outcome: | The results show that the robust use of both knowledge sources is an emergent property . the results provide guidance for designing training data that supports the reliability of parametric and in-context knowledge in language models. |
Locally Differentially Private In-Context Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Large pretrained language models (LLMs) have shown surprising In-Context Learning ability. |
| Approach: | They propose a locally differentially private framework of in-context learning for LLMs that can be augmented with a private database for some specific task. |
| Outcome: | The proposed framework can predict labels without additional parameter modifications without input-label pairs . |
Knowledge Unlearning for Mitigating Privacy Risks in Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work shows that an adversary can extract training data from Pretrained Language Models including Personally Identifiable Information (PII) such as names, phone numbers, and email addresses. |
| Approach: | They propose to use knowledge unlearning to reduce privacy risks for LMs by performing gradient ascent on target token sequences instead of trying to unlearn all the data at once. |
| Outcome: | The proposed method can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust. |
CAPE: Context-Aware Private Embeddings for Private Language Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to obtain text representations or embeddings with these models encoding personally identifiable information may lead to privacy leaks. |
| Approach: | They propose a novel approach which combines differential privacy and adversarial learning to preserve privacy during training of embeddings. |
| Outcome: | The proposed approach reduces private information leakage by 3% over the current method. |
Unintended Memorization of Sensitive Information in Fine-Tuned Language Models (2026.eacl-long)
Copied to clipboard
Marton Szep, Jorge Marin Ruiz, Georgios Kaissis, Paulina Seidl, Rüdiger von Eisenhart-Rothe, Florian Hinterwimmer, Daniel Rueckert
| Challenge: | Large Language Models (LLMs) on sensitive datasets carry a substantial risk of unintended memorization and leakage of Personally Identifiable Information (PII) prior studies have analyzed memorizing dynamics in LLMs during pre-training and fine-tuning. |
| Approach: | They investigate the vulnerability of PII that appears only in model inputs, not in training targets. |
| Outcome: | The proposed methods show that post-training methods provide more consistent privacy-utility trade-offs . |
Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Fine-tuning of frontier models can lead to privacy collapse, causing optimisation for helpfulness, exposure to user information, and debugging code printing internal variables. |
| Approach: | They propose to fine-tune frontier models to adapt to specific domains and align with organizational workflows and user preferences. |
| Outcome: | The proposed model fails to perform on safety and utility benchmarks while exhibiting severe privacy vulnerabilities. |
MoPe: Model Perturbation based Privacy Attacks on Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that Large Language Models can unintentionally leak sensitive information . a new method to identify with high confidence if a given text is in training data is proposed . |
| Approach: | They propose a method to detect if a given text is in a pre-trained language model . they show that MoPe is more effective than existing loss-based attacks . |
| Outcome: | The proposed method is more effective than loss-based attacks and perturbation-based methods. |