Papers with DP-SGD
DP-NMT: Scalable Differentially Private Machine Translation (2024.eacl-demo)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) is a popular text generation task, yet there is nagging data privacy concerns. |
| Approach: | They propose an open-source framework for a privacy-preserving NMT with DP-SGD. |
| Outcome: | The proposed framework is open-source and open to the public . it combines models, datasets, and evaluation metrics to demonstrate its effectiveness. |
Differentially Private Next-Token Prediction of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are becoming increasingly important for ensuring privacy, but DP-SGD overestimates an adversary’s capabilities in having white box access. |
| Approach: | They propose a private prediction protocol for next-token prediction that utilizes the inherent stochasticity of next-toning and a public model to achieve Differential Privacy. |
| Outcome: | The proposed protocol outperforms DP-SGD and DP training methods for privacy on large datasets. |
Differentially Private Language Models for Secure Data Sharing (2022.emnlp-main)
Copied to clipboard
| Challenge: | a variety of deanonymization attacks allow the re-identification of individuals from tabular data. |
| Approach: | They propose to train a language model in a differentially private manner and sample data from it . they find that the model generates fluent textual datasets with privacy guarantees . |
| Outcome: | The proposed methods outperform direct classifiers with DP-SGD in the real-world. |
Large-Scale Differentially Private BERT (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that scaling up the batch size to millions improves the utility of a DP-SGD step for BERT. |
| Approach: | They propose to use differentially private SGD to pretrain BERT-Large with a batch size of millions to improve the utility of the DP-SGD step. |
| Outcome: | The proposed approach achieves a masked language model accuracy of 60.5% at a batch size of 2M, which is a reasonable privacy setting. |
One size does not fit all: Investigating strategies for differentially-private learning across NLP tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on the efficiency of differentially-private stochastic gradient descent (DP-SGD) in NLP is inconclusive or even counter-intuitive. |
| Approach: | They propose to use differentially-private stochastic gradient descent (DP-SGD) to preserve privacy in NLP by using modern neural models based on BERT and XtremeDistil architectures to conduct extensive experiments. |
| Outcome: | The proposed models and training strategies provide the best trade-off between privacy and performance on different NLP tasks. |