Machine Learning Models that Remember Too Much

By Congzheng Song, Thomas Ristenpart, Vitaly Shmatikov. 2017

597 citations recorded by the source corpus.

Machine learning (ML) is becoming a commodity. Numerous ML frameworks and services are available to data holders who are not ML experts but want to train predictive models on their data. It is important that ML models trained on sensitive inputs (e.g., personal images or documents) not leak too much information about the training data. We consider a malicious ML provider who supplies model-training code to the data holder, does \emph{not} observe the training, but then obtains white- or black-box access to the resulting model. In this setting, we design and implement practical algorithms, some of them very similar to standard ML techniques such as regularization and data augmentation, that "memorize" information about the training dataset in the model\textemdash yet the model is as accurate and predictive as a conventionally trained model. We then explain how the adversary can extract memorized information from the model. We evaluate our techniques on standard ML tasks for image classification (CIFAR10), face recognition (LFW and FaceScrub), and text analysis (20 Newsgroups and IMDB). In all cases, we show how our algorithms create models that have high predictive power yet allow accurate extraction of subsets of their training data.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Ethical Challenges in Data-Driven Dialogue Systems (2017, background): “It has been shown that model-inversion attacks can be used to compromise the privacy of training data through information leakage in machine learning models [38].”
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, neutral): “[44] also study training data extraction.”
How To Backdoor Federated Learning (2018, background): “ow well the model has learned its main task but they don’t measure what else the model has learned. This extra capacity can be used to memorize random information [42] or abused to leak training data [38] or, as we show in this paper, to introduce covert backdoor functionality without any signiﬁcant”
Salvaging Federated Learning by Local Adaptation (2020, background): “ML models can leak their training data (Song et al., 2017; Shokri et al., 2017).”
Theoretical Guarantees for Model Auditing with Finite Adversaries (2019, background): “Indeed, recent works show that “overparameterized” machine learning models have the tendency to memorize unique training examples and overlearn sensitive attributes [1–9].”
Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, methodology): “Attacks targeting data privacy include: the adversary inferring whether input examples were used to train the target model with membership inference attacks [37, 47, 64], learning global properties of training data with property inference attacks [12], or covert channel model training attacks [52].”
Exploiting Unintended Feature Leakage in Collaborative Learning (2018, methodology): “[55] engineer an ML model that memorizes the training data, which can then be extracted with black-box access to the model.”
Contamination Attacks and Mitigation in Multi-Party Machine Learning (2018, background): “Each party can review the code that will be used to train the model to ensure no backdoors are present (e.g., to prevent attacks described in Song et al. [41]).”
Property Inference Attacks on Fully Connected Neural Networks using Permutation Invariant Representations (2018, background): “This assumption is reasonable and quite common nowadays [5, 25, 39].”
Stealing Hyperparameters in Machine Learning (2018, background): “Several studies [45], [48], [42] demonstrated even stronger attacks, i.e., an attacker can infer whether a particular instance was in the training dataset or not.”
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks (2019, background): “Membership attacks [28, 15, 24, 14, 26] attempt to determine whether a particular cohort of data was used for training, without any constructive indication on how to actively forget it.”
Auditing Data Provenance in Text-Generation Models (2018, neutral): “[29] present algorithms that intentionally encode the training data in the model.”
Large image datasets: A pyrrhic win for computer vision? (2020, positive): “In the context of face recognition, works such as [74] have demonstrated that CNNs with high predictive power unwittingly accommodate accurate extraction of subsets of the facial images that they were trained on, thus abetting dataset leakage.”
LOGAN: Membership Inference Attacks Against Generative Models (2017, background): “[58] attacks force a machine learning model to memorize the training data in such a way that an adversary can later extract training inputs with only black-box access to the model.”
Federated Meta-Learning with Fast Convergence and Efficient Communication (2018, background): “In the future, we will explore the following directions: (1) We want to study whether the FedMeta framework has additional advantages in preserving user privacy from the model attack perspective [26, 24, 25, 22], as the global model shared in the current federated learning approaches still includes ”
Differentially Private Model Publishing for Deep Learning (2019, background): “[36] proposed training phase attacks which perform minor modifications to training algorithms to make them output models which encode a significant amount of information about the training dataset while achieving high quality metrics like accuracy and generalizability.”
Understanding Membership Inferences on Well-Generalized Learning Models (2018, background): “ed models when the adversary has control over the training algorithm. Speciﬁcally, the adversary can encode private information of the training dataset into the predictions of well-generalized models [33]. These two attacks [32,33] can be formalized under a uniform theoretical framework [39]. The ri”
Model inversion attacks against collaborative inference (2019, background): “The third type are model encoding attacks [42]: the adversary with direct access to the training data can encode the sensitive data into the model for a receiver entity to retrieve.”