Training Production Language Models without Memorizing User Data

By Swaroop Indra Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H. Brendan McMahan, Franccoise Beaufays. 2020

97 citations recorded by the source corpus.

This paper presents the first consumer-scale next-word prediction (NWP) model trained with Federated Learning (FL) while leveraging the Differentially Private Federated Averaging (DP-FedAvg) technique. There has been prior work on building practical FL infrastructure, including work demonstrating the feasibility of training language models on mobile devices using such infrastructure. It has also been shown (in simulations on a public corpus) that it is possible to train NWP models with user-level differential privacy using the DP-FedAvg algorithm. Nevertheless, training production-quality NWP models with DP-FedAvg in a real-world production environment on a heterogeneous fleet of mobile phones requires addressing numerous challenges. For instance, the coordinating central server has to keep track of the devices available at the start of each round and sample devices uniformly at random from them, while ensuring \emph{secrecy of the sample}, etc. Unlike all prior privacy-focused FL work of which we are aware, for the first time we demonstrate the deployment of a differentially private mechanism for the training of a production neural network in FL, as well as the instrumentation of the production training infrastructure to perform an end-to-end empirical measurement of unintended memorization.

Subfields: Artificial Intelligence, Computer Science Applications, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Ethical and social risks of harm from Language Models (2021, methodology): “Work toward such robustness focuses on algorithmic tools used during the training of the LM, such as differential privacy methods (Abadi et al., 2016; Ramaswamy et al., 2020).”
Quantifying Memorization Across Neural Language Models (2022, methodology): “Because existing approaches for provably-correct privacy-preserving training of machine learning are applied only rarely in practice (Abadi et al., 2016; Thakkar et al., 2020; Ramaswamy et al., 2020), it is common to attempt post-hoc privacy auditing (Jayaraman and Evans, 2019; Jagielski et al.”
Extracting Training Data from Large Language Models (2020, background): “In the content of language modeling, recent work demonstrates the privacy benefits of user-level DP models [52].”
Practical and Private (Deep) Learning without Sampling or Shuffling (2021, background): “Federated Learning (FL) and DP-FTRL: There has been prior work (Balle et al., 2020; Ramaswamy et al., 2020) detailing challenges for obtaining strong privacy guarantees that incorporate limited availability of participating clients in real-world applications of Federated Learning.”
What Does it Mean for a Language Model to Preserve Privacy? (2022, background): “To address these privacy concerns, there is a growing body of literature that aims to create privacypreserving language models [64, 2, 56, 98, 84, 40, 79].”
Taxonomy of Risks posed by Language Models (2022, methodology): “One approach to preventing privacy leaks is to apply algorithmic tools such as differential privacy methods during LM training [1, 153].”
Submix: Practical Private Prediction for Large-Scale Language Models (2022, methodology): “Indeed, existing work only performs DP-SGD training of small feedforward networks (Kerrigan et al., 2020) and RNNs (McMahan et al., 2017; Ramaswamy et al., 2020), often with an unsatisfactory privacy-utility trade-off.”
Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy (2022, background): “…regurgitate potentially private information, like phone numbers and addresses, as well as memorize long sequences from their training sets (Carlini et al., 2019, 2021; Lee et al., 2021; Carlini et al., 2022; Zhang et al., 2021; Thakkar et al., 2021; Ramaswamy et al., 2020; Kandpal et al., 2022).”
Large Language Models Can Be Strong Differentially Private Learners (2021, methodology): “Ramaswamy et al. (2020) train production grade next-word prediction models using DP-FedAvg with millions of users.”
Differentially Private Fine-tuning of Language Models (2021, background): “Though pre-training on public data is suggested (McMahan et al., 2018), public data appears to only be used in one of these works for honest hyperparameter selection (Ramaswamy et al., 2020).”