Papers by Rohan Anil
Large-Scale Differentially Private BERT (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that scaling up the batch size to millions improves the utility of a DP-SGD step for BERT. |
| Approach: | They propose to use differentially private SGD to pretrain BERT-Large with a batch size of millions to improve the utility of the DP-SGD step. |
| Outcome: | The proposed approach achieves a masked language model accuracy of 60.5% at a batch size of 2M, which is a reasonable privacy setting. |