Understanding Unintended Memorization in Federated Learning

By Om Thakkar, Swaroop Indra Ramaswamy, Rajiv Mathews, Franccoise Beaufays. 2020

49 citations recorded by the source corpus.

Recent works have shown that generative sequence models (e.g., language models) have a tendency to memorize rare or unique sequences in the training data. Since useful models are often trained on sensitive data, to ensure the privacy of the training data it is critical to identify and mitigate such unintended memorization. Federated Learning (FL) has emerged as a novel framework for large-scale distributed learning tasks. However, it differs in many aspects from the well-studied central learning setting where all the data is stored at the central server. In this paper, we initiate a formal study to understand the effect of different components of canonical FL on unintended memorization in trained models, comparing with the central learning setting. Our results show that several differing components of FL play an important role in reducing unintended memorization. Specifically, we observe that the clustering of data according to users---which happens by design in FL---has a significant effect in reducing such memorization, and using the method of Federated Averaging for training causes a further reduction. We also show that training with a strong user-level differential privacy guarantee results in models that exhibit the least amount of unintended memorization.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Quantifying Memorization Across Neural Language Models (2022, methodology): “…provably-correct approaches for privacy-preserving training of machine learning models are applied only rarely in practice (Abadi et al., 2016; Thakkar et al., 2020; Ramaswamy et al., 2020), it is common to attempt post-hoc privacy auditing (Jayaraman and Evans, 2019; Jagielski et al., 2020;…”
Extracting Training Data from Large Language Models (2020, background): “, for more epochs than typical) [6, 61, 63], settings with white-box model access [45], or settings where the adversary has a priori knowledge of the secret they want to extract (e.”
Practical and Private (Deep) Learning without Sampling or Shuffling (2021, methodology): “We emphasize that in the context of ML applications, using a DP mechanism even with a large ε is practically much better for privacy than using a non-DP mechanism (Song & Shmatikov, 2019; Jagielski et al., 2020; Thakkar et al., 2020; Nasr et al., 2021).”
Counterfactual Memorization in Neural Language Models (2021, methodology): “…work on memorization in neural language models demonstrated the ability to extract memorized training data, including sensitive data such as phone numbers and usernames [Carlini et al., 2020, Ziegler, 2021, Carlini et al., 2019, Henderson et al., 2017, Thakkar et al., 2020, Thomas et al., 2020].”
Measuring Forgetting of Memorized Training Examples (2022, background): “Resulting privacy concerns have led to a variety of techniques for private machine learning, including differentially private training [1, 61, 62, 23, 53], machine unlearning [5, 58, 67], and various heuristics like regularization [56], data augmentation [2] or gradient clipping [10, 72, 32].”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, background): “Existing work focuses on analyzing memorization post-training [8, 9, 10, 11].”