Large Language Models Can Be Strong Differentially Private Learners

By Xuechen Li, Florian Tramèr, Percy Liang, Tatsunori B. Hashimoto. 2021

533 citations recorded by the source corpus.

Differentially Private (DP) learning has seen limited success for building large deep learning models of text, and straightforward attempts at applying Differentially Private Stochastic Gradient Descent (DP-SGD) to NLP tasks have resulted in large performance drops and high computational overhead. We show that this performance drop can be mitigated with (1) the use of large pretrained language models; (2) non-standard hyperparameters that suit DP optimization; and (3) fine-tuning objectives which are aligned with the pretraining procedure. With the above, we obtain NLP models that outperform state-of-the-art DP-trained models under the same privacy budget and strong non-private baselines -- by directly fine-tuning pretrained models with DP optimization on moderately-sized corpora. To address the computational challenge of running DP-SGD with large Transformers, we propose a memory saving technique that allows clipping in DP-SGD to run without instantiating per-example gradients for any linear layer in the model. The technique enables privately training Transformers with almost the same memory cost as non-private training at a modest run-time overhead. Contrary to conventional wisdom that DP optimization fails at learning high-dimensional models (due to noise that scales with dimension) empirical results reveal that private learning with pretrained language models doesn't tend to suffer from dimension-dependent performance degradation. Code to reproduce results can be found at https://github.com/lxuechen/private-transformers.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “, 2022a) Privacy Differential Private training (Kerrigan et al., 2020; Li et al., 2022; Shi et al., 2021); Knowledge Unlearning (Jang et al.”
Foundation Models and Fair Use (2023, background): “Recent works have developed substantial improvements to address this drawback through the use of better software primitives (Anil et al., 2021) and training techniques (Li et al., 2021; Yu et al., 2021; De et al., 2022; Sander et al., 2022).”
What Does it Mean for a Language Model to Preserve Privacy? (2022, background): “To address these privacy concerns, there is a growing body of literature that aims to create privacypreserving language models [64, 2, 56, 98, 84, 40, 79].”
Differentially Private Decoding in Large Language Models (2022, background): “Recent works (Li et al., 2021; Yu et al., 2021; Dupuy et al., 2021) address the aforementioned challenges, but DP training still remains an active area of interest as no conclusive method satisfactorily balances all aspects.”
Unlocking High-Accuracy Differentially Private Image Classification through Scale (2022, methodology): “Intriguingly, previous work has observed that wide ranges of the clipping norm 𝐶 can provide near optimal performance provided that (i) the clipping norm is small enough, and (ii) the learning-rate is re-scaled accordingly (Kurakin et al., 2022; Li et al., 2021).”
Are Large Pre-Trained Language Models Leaking Your Personal Information? (2022, methodology): “Deﬁnition 2 (Association) Personal information x can be associated by a model f if there exists a prompt p (usually containing the information owner’s name) designed by the attacker (who does not have access to the training data) that can prompt f to produce x using greedy decoding.”
Knowledge Unlearning for Mitigating Privacy Risks in Language Models (2022, background): “…data (data preprocessing) (Aura et al., 2006; Dernoncourt et al., 2017; Lison et al., 2021; Kandpal et al., 2022) or attempt to design algorithms that ensure differential privacy (DP) (Dwork, 2008; Dwork et al., 2006; Abadi et al., 2016; Anil et al., 2021; Li et al., 2022; Yu et al., 2022).”
Provably Confidential Language Modelling (2022, methodology): “We run the experiment for the GPT-2 model following Li et al. (2021) 6 , in which they propose ghost clipping method to alleviate the computational challenge of running DP-SGD with large Transformers.”
Differentially Private Fine-tuning of Language Models (2021, methodology): “In a concurrent work, Li et al. (2022) also investigate DP ﬁne-tuning of LLMs.”
Opacus: User-Friendly Differential Privacy Library in PyTorch (2021, methodology): “[9] generalized the Goodfellow method to handle sequential inputs, making fine-tuning large Transformers under DP computationally efficient.”
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy (2023, methodology): “However, Li et al. (2022b) highlight that for such a strategy to work well, the pretraining and fine-tuning tasks should have similar objectives.”
Analyzing Leakage of Personally Identifiable Information in Language Models (2023, methodology): “[35] propose an efficient method for differential-privately fine-tuning LMs on private data.”
Security and Privacy Challenges of Large Language Models: A Survey (2024, unknown): “Also, the high cost of accessing commercialized LLMs, such as GPT-3.”
Submix: Practical Private Prediction for Large-Scale Language Models (2022, methodology): “For DP-SGD, concurrent work (Li et al., 2021) proposed training with very large batch size and ﬁxed number of updates to signiﬁcantly improve privacy-utility trade-off.”
Membership Inference Attacks against Language Models via Neighbourhood Comparison (2023, methodology): “To test the effectiveness of differential privacy as a defense against neighbourhood attacks, we follow Li et al. (2022) and train our target model GPT-2 in a differentially private manner on AG News, where our attack performed the best.”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, background): “Memorization in Language Models: Unintended memorization is a known challenge for language models [12, 13], which makes them open to extraction attacks [14, 15] and membership inference attacks [16, 17], although there has been work on mitigating these vulnerabilities [11, 18].”