Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks

By Milad Nasr, Reza Shokri, Amir Houmansadr. 2018

293 citations recorded by the source corpus.

Deep neural networks are susceptible to various inference attacks as they remember information about their training data. We perform a comprehensive analysis of white-box privacy inference attacks on deep learning models. We measure the privacy leakage by leveraging the final model parameters as well as the parameter updates during the training and fine-tuning processes. We design the attacks in the stand-alone and federated settings, with respect to passive and active inference attackers, and assuming different adversary prior knowledge. We design and evaluate our novel white-box membership inference attacks against deep learning algorithms to measure their training data membership leakage. We show that a straightforward extension of the known black-box attacks to the white-box setting (through analyzing the outputs of activation functions) is ineffective. We therefore design new algorithms tailored to the white-box setting by exploiting the privacy vulnerabilities of the stochastic gradient descent algorithm, widely used to train deep neural networks. We show that even well-generalized models are significantly susceptible to white-box membership inference attacks, by analyzing state-of-the-art pre-trained and publicly available models for the CIFAR dataset. We also show how adversarial participants of a federated learning setting can run active membership inference attacks against other participants, even when the global model achieves high prediction accuracies.

Subfields: Federated & Distributed Learning, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Federated & Distributed Learning

In Privacy & Security

Citations in the corpus

How To Backdoor Federated Learning (2018, background): “Secure aggregation of model updates [7] is essential for privacy because model updates leak sensitive information about participants’ training data [45, 48].”
Central Server Free Federated Learning over Single-sided Trust Social Networks (2019, background): “These properties create difficulties for many attacking methods, such as [43].”
Bypassing Backdoor Detection Algorithms in Deep Learning (2019, background): “Having access to the model predictions or parameters, an adversary can infer sensitive information about the model’s training set [20], [18].”
A Hybrid Approach to Privacy-Preserving Federated Learning (2018, background): “That is, participant Pi should not be able to infer information about D j when i , j as shown in [30].”
Information Leakage in Embedding Models (2020, background): “This subsequently led to a large body of work exploring this space [45, 48, 60, 61, 77].”
DarkneTZ: towards model privacy at the edge using trusted execution environments (2020, methodology): “ that the majority of smartphone applications (70.6%) that use deep learning are for image processing [55]. Moreover, the state of the art MIA we are considering is demonstrated against such datasets [38]. CIFAR-100 includes 50k training and 10k test images of size 32 ×32 ×3 belonging to 100 classes”
Enhancing Privacy via Hierarchical Federated Learning (2020, methodology): “[20] use gradient ascent to perform a membership inference attack.”
Privacy preservation in federated learning: An insightful survey from the GDPR perspective (2020, background): “dataset8 showing a high accuracy of 74% compared to only 50% in passive attack [93, 94].”
Federated Imitation Learning: A Privacy Considered Imitation Learning Framework for Cloud Robotic Systems with Heterogeneous Sensor Data (2019, background): “Federated learning was first proposed in [26], which showed its effectiveness through experiments on various datasets.”
The Limitations of Federated Learning in Sybil Settings (2020, background): “Model free riding [34] Membership inference [41,42,54]”
Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems (2019, methodology): “In federated learning systems, the raw data is collected and stored at multiple edge nodes, and a machine learning model is trained from the distributed data without sending the raw data from the nodes to a central place [23], [24].”
Biscotti: A Blockchain System for Private and Secure Federated Learning (2021, background): “By observing a peer’s model updates from each verification round, an adversary can perform an information leakage attack [13], [20], [51], [52], [53], [54] and recover details about a victim’s training data.”
LDP-Fed: federated learning with local differential privacy (2020, background): “type of encoding of each participant’s private data. Recent work has indeed demonstrated that effective membership inference privacy attacks may be launched given access to these shared model updates [16]. Additionally, the final model itself will also leak with prediction outputs (leakage point 3) ”
TiFL: A Tier-based Federated Learning System (2020, methodology): “FL has been used together with privacy preserving approaches such as differential privacy to prevent attacks that aim to extract private information [22, 26].”
Disparate Vulnerability: on the Unfairness of Privacy Attacks Against Machine Learning (2019, background): “A series of works have shown the feasibility of MIA when the model is offered as a service [26], in federated settings [22]; moreover, the attacks can be effective even when the model is well generalized [19].”
Eavesdrop the Composition Proportion of Training Labels in Federated Learning (2019, background): “importing backdoor to FL [14, 15], adversarial poisoning [16, 17, 18], membership or property inference [19, 20, 21, 22], and reconstruction attack [23, 24, 25].”