Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning

By Milad Nasr, Reza Shokri, Amir Houmansadr. 2018

1,858 citations recorded by the source corpus.

Deep neural networks are susceptible to various inference attacks as they remember information about their training data. We design white-box inference attacks to perform a comprehensive privacy analysis of deep learning models. We measure the privacy leakage through parameters of fully trained models as well as the parameter updates of models during training. We design inference algorithms for both centralized and federated learning, with respect to passive and active inference attackers, and assuming different adversary prior knowledge. We evaluate our novel white-box membership inference attacks against deep learning algorithms to trace their training data records. We show that a straightforward extension of the known black-box attacks to the white-box setting (through analyzing the outputs of activation functions) is ineffective. We therefore design new algorithms tailored to the white-box setting by exploiting the privacy vulnerabilities of the stochastic gradient descent algorithm, which is the algorithm used to train deep neural networks. We investigate the reasons why deep learning models may leak information about their training data. We then show that even well-generalized models are significantly susceptible to white-box membership inference attacks, by analyzing state-of-the-art pre-trained and publicly available models for the CIFAR dataset. We also show how adversarial participants, in the federated learning setting, can successfully run active membership inference attacks against other participants, even when the global model achieves high prediction accuracies.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “Neural machine learning methods (Nasr et al., 2019; Shokri et al., 2017), and language models in particular (Carlini et al., 2019; 2020) can be susceptible to training data leakage, where sensitive information can be extracted from the models.”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “Similar privacy problems are observed for LLMs (e.g. Nasr et al., 2019; Shokri et al., 2017; Carlini et al., 2019, 2020).”
Extracting Training Data from Large Language Models (2020, background): “At the same time, machine learning models are known to leak information about their (potentially private) training data—both in general [42, 60] and in the specific case of language models [6, 40].”
Red Teaming Language Models with Language Models (2022, methodology): “Data leakage can be used to infer the data used in training (“membership inference”; Shokri et al., 2017; Song and Shmatikov, 2019; Nasr et al., 2019; Hisamoto et al., 2020; Carlini et al., 2021), helping adversaries clone private, commercial LMs, violating intellectual property (Ateniese et al.,…”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “At the same time, datasets curated from the Web could come with risks such as unsuitable content (adult content, profanity) and unintended privacy leakage [57, 17, 18].”
Transformer Feed-Forward Layers Are Key-Value Memories (2020, background): “…the pattern-identiﬁcation process; memory cells might affect training-data privacy as they could facilitate white-box membership inference (Nasr et al., 2019); and studying cases where a correct pattern is identiﬁed but then suppressed during aggregation may guide architectural novelties.”
Privacy Amplification by Decentralization (2020, background): “In fact, FL provides an additional attack surface as the participants share intermediate updates (Nasr et al., 2019; Geiping et al., 2020).”
FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “…challenge, existing research focuses on developing new adversarial attack and defense techniques to make federated learning robust [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 22], and proposing methods such as differential privacy (DP) and secure multiparty computation (SMPC) to protect…”
Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, methodology): “model’s training data or the target model. Attacks targeting data privacy include: the adversary inferring whether input examples were used to train the target model with membership inference attacks [37, 47, 64], learning global properties of training data with property inference attacks [12], or c”
What Does it Mean for a Language Model to Preserve Privacy? (2022, background): “These attacks can be seen as privacy risk analysis tools [67], which help reveal how much the model has memorized the individual samples in its training set, and what the risk of individual users is [85, 69, 59, 81, 88].”
A Study of Face Obfuscation in ImageNet (2021, background): “Membership inference attack infers whether an individual was included in training (Shokri et al., 2017; Nasr et al., 2019; Hisamoto et al., 2020).”
Federated Learning for Internet of Things: Recent Advances, Taxonomy, and Open Challenges (2020, background): “devices private information [33], [103].”
Systematic Evaluation of Privacy Risks of Machine Learning Models (2020, methodology): “Furthermore, in the case where the adversary knows a subset of target model’s training set and test set, the attack classifier can be directly trained with target model’s predictions on those known samples, and then tested on unknown training and test sample [29, 30].”
A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, methodology): “For example, membership inference attacks [125], [154] can infer whether a specific data record is used in the training.”
When Machine Unlearning Jeopardizes Privacy (2020, background): “Since then, membership inference has been extensively investigated in various ML models and tasks, such as federated learning [46], white-box classification [48], generative adversarial networks [13, 28], natural language processing [67], and computer vision segmentation [30].”
Zero-Shot Machine Unlearning (2022, background): “Thereafter, several types of membership inference attacks have been proposed [25], [28], [29], [30], [31].”
Mobile Edge Computing, Blockchain and Reputation-based Crowdsourcing IoT Federated Learning: A Secure, Decentralized and Privacy-preserving System (2019, background): “[25] performed a comprehensive privacy analysis with white-box inference attacks.”