Differentially Private Language Models Benefit from Public Pre-training
By Gavin Kerrigan, Dylan Slack, Jens Tuyls. 2020
71 citations recorded by the source corpus.
Language modeling is a keystone task in natural language processing. When training a language model on sensitive information, differential privacy (DP) allows us to quantify the degree to which our private data is protected. However, training algorithms which enforce differential privacy often lead to degradation in model quality. We study the feasibility of learning a language model which is simultaneously high-quality and privacy preserving by tuning a public base model on a private corpus. We find that DP fine-tuning boosts the performance of language models in the private domain, making the training of such models possible.
Subfields: Large Language Models, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Large Language Models
- James Y. Zou
- Vidhisha Balachandran
- Chuan Guo
- Yulia Tsvetkov
- Percy Liang
- L. V. D. van der Maaten
- Tatsunori B. Hashimoto
- Xuechen Li
In Safety, Robustness & Interpretability
- Percy Liang
- Yulia Tsvetkov
- James Y. Zou
- Vidhisha Balachandran
- Florian Tramèr
- Tatsunori B. Hashimoto
- Sachin Kumar
- Chuan Guo
Citations in the corpus
- Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “capabilities, most prominent solutions are based on differentially private (DP) learning (Kerrigan et al., 2020; Shi et al., 2021).”
- Large Language Models Can Be Strong Differentially Private Learners (2021, background): “R EFFECTS OF VARYING AND δ ON DP FULL FINE-TUNING PERFORMANCE Figure 9 shows how different values of and δ affect the performance of full fine-tuning under DP.”
- Submix: Practical Private Prediction for Large-Scale Language Models (2022, background): “Indeed, existing work only performs DP-SGD training of small feedforward networks (Kerrigan et al., 2020) and RNNs (McMahan et al., 2017; Ramaswamy et al., 2020), often with an unsatisfactory privacy-utility trade-off.”