Improving Privacy Guarantee and Efficiency of Latent Dirichlet Allocation Model Training Under Differential Privacy (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Latent Dirichlet allocation (LDA) is a widely used topic model to discover the latent semantic of text data. |
| Approach: | They propose to combine a subsampling method with CGS to improve efficiency while amplifying privacy by using a novel metric, the efficiency–privacy function. |
| Outcome: | The proposed algorithm improves efficiency while amplifying privacy while subsampling in CGS increases efficiency while preserving privacy. |
Similar Papers
LDEDE: LRP-Driven Efficient Detection and Editing Framework for LLM Privacy Neurons (2026.findings-acl)
Copied to clipboard
Zhao Zhengyuan, Cao Lifeng, null Sunhaodong, Shi Haotian, Du Xuehui, Liu Aodi, Niu Lanjie, Yang Xiaocheng
| Challenge: | Existing privacy protection methods fail to cover context-dependent sensitive information and are prone to performance degradation. |
| Approach: | They propose a Layer-wise Relevance Propagation-driven framework for efficient privacy neuron detection and editing. |
| Outcome: | The proposed framework achieves 80% higher efficiency than gradient attribution methods while reducing leakage risks of Phone, Email, and medical privacy by 42.7%–73.5% on average and cutting computational time by 60%–90%. |
Fine-Tuning Language Models with Differential Privacy through Adaptive Noise Allocation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models can memorize detailed information and patterns, but raise privacy concerns . ANADP reduces the performance gap between regular and DP fine-tuning while maintaining the privacy constraints. |
| Approach: | They propose an algorithm that allocates additive noise based on the importance of model parameters to reduce the performance gap between regular fine-tuning and traditional DP fine- tuning. |
| Outcome: | The proposed algorithm narrows the performance gap between regular fine-tuning and traditional DP fine- tuning while maintaining privacy constraints. |
One size does not fit all: Investigating strategies for differentially-private learning across NLP tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on the efficiency of differentially-private stochastic gradient descent (DP-SGD) in NLP is inconclusive or even counter-intuitive. |
| Approach: | They propose to use differentially-private stochastic gradient descent (DP-SGD) to preserve privacy in NLP by using modern neural models based on BERT and XtremeDistil architectures to conduct extensive experiments. |
| Outcome: | The proposed models and training strategies provide the best trade-off between privacy and performance on different NLP tasks. |
Privacy Regularization: Joint Privacy-Utility Optimization in LanguageModels (2021.naacl-main)
Copied to clipboard
Fatemehsadat Mireshghallah, Huseyin Inan, Marcello Hasegawa, Victor Rühle, Taylor Berg-Kirkpatrick, Robert Sim
| Challenge: | Neural language models have a high capacity for memorization of training samples . however, this can cause privacy degradation and disparate impact on subgroups of users . |
| Approach: | They propose two privacy-preserving regularization methods for training language models that enable joint optimization of utility and privacy. |
| Outcome: | The proposed methods have favorable utility-privacy trade-off, faster training and uniform treatment of under-represented subgroups. |
Granularity is crucial when applying differential privacy to text: An investigation for neural machine translation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Differential privacy (DP) is a method that is increasingly popular in NLP but the choice of granularity at which it is applied is often neglected. |
| Approach: | They propose to apply differential privacy (DP) to a neural machine translation dataset to protect individual data points. |
| Outcome: | The proposed method is more resistant to membership inference attacks than the document-level NMT system. |
Federated Learning of Gboard Language Models with Differential Privacy (2023.acl-industry)
Copied to clipboard
Zheng Xu, Yanxiang Zhang, Galen Andrew, Christopher Choquette, Peter Kairouz, Brendan Mcmahan, Jesse Rosenstock, Yuanbo Zhang
| Challenge: | Using federated learning and differential privacy, we train and deploy language models with federation and DP in Google Keyboard. |
| Approach: | They train and deploy language models with federated learning and differential privacy in Google Keyboard . |
| Outcome: | The proposed algorithm achieves meaningfully formal DP guarantees without uniform sampling of clients. |
How reparametrization trick broke differentially-private text representation learning (2022.acl-short)
Copied to clipboard
| Challenge: | Differential privacy (DP) is a formal mathematical treatment of privacy protection . it guarantees how much privacy can be lost in the worst case . adapting DP mechanisms to NLP properly is largely non-trivial task . |
| Approach: | They propose to use differential privacy to learn text representations using DPText to quantify and guarantee how much privacy can be lost in the worst case. |
| Outcome: | The proposed methods are falsely claimed to be differentially private and violate privacy loss guarantees. |
Training Text-to-Text Transformers with Privacy Guarantees (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in NLP often stem from large transformer-based pre-trained models. |
| Approach: | They propose differentially private (DP) training as a potential mitigation for models that can memorize parts of training data. |
| Outcome: | The proposed model can memorize parts of training data and mitigate memorization concerns. |
Locally Differentially Private In-Context Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Large pretrained language models (LLMs) have shown surprising In-Context Learning ability. |
| Approach: | They propose a locally differentially private framework of in-context learning for LLMs that can be augmented with a private database for some specific task. |
| Outcome: | The proposed framework can predict labels without additional parameter modifications without input-label pairs . |
Learning and Evaluating a Differentially Private Pre-trained Language Model (2021.findings-emnlp)
Copied to clipboard
Shlomo Hoory, Amir Feder, Avichai Tendler, Sofia Erell, Alon Peled-Cohen, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayelet Benjamini, Avinatan Hassidim, Yossi Matias
| Challenge: | Contextual language models have improved performance but can lead to information leakage . |
| Approach: | They propose a differentially-private word-piece algorithm that allows training a tailored domain-specific vocabulary while maintaining privacy. |
| Outcome: | The proposed model can guarantee privacy while maintaining good model performance. |