Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning
By Briland Hitaj, Giuseppe Ateniese, Fernando Perez-Cruz. Conference on Computer and Communications Security, 2017
1,613 citations recorded by the source corpus.
Deep Learning has recently become hugely popular in machine learning for its ability to solve end-to-end learning systems, in which the features and the classifiers are learned simultaneously, providing significant improvements in classification accuracy in the presence of highly-structured and large databases. Its success is due to a combination of recent algorithmic breakthroughs, increasingly powerful computers, and access to significant amounts of data. Researchers have also considered privacy implications of deep learning. Models are typically trained in a centralized manner with all the data being processed by the same training algorithm. If the data is a collection of users' private data, including habits, personal pictures, geographical positions, interests, and more, the centralized server will have access to sensitive information that could potentially be mishandled. To tackle this problem, collaborative deep learning models have recently been proposed where parties locally train their deep learning structures and only share a subset of the parameters in the attempt to keep their respective training sets private. Parameters can also be obfuscated via differential privacy (DP) to make information extraction even more challenging, as proposed by Shokri and Shmatikov at CCS'15. Unfortunately, we show that any privacy-preserving collaborative deep learning is susceptible to a powerful attack that we devise in this paper. In particular, we show that a distributed, federated, or decentralized deep learning approach is fundamentally broken and does not protect the training sets of honest participants. The attack we developed exploits the real-time nature of the learning process that allows the adversary to train a Generative Adversarial Network (GAN) that generates prototypical samples of the targeted training set that was meant to be private (the samples generated by the GAN are intended to come from the same distribution as the training data). Interestingly, we show that record-level differential privacy applied to the shared parameters of the model, as suggested in previous work, is ineffective (i.e., record-level DP is not designed to address our attack).
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Federated & Distributed Learning, Privacy & Security
View Paper: https://arxiv.org/abs/1702.07464
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Qiang Yang
- Reza Shokri
- Prateek Mittal
- Philip S. Yu
- Vitaly Shmatikov
- Bernt Schiele
- Yang Zhang
- Mario Fritz
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Bernt Schiele
- Ming-Hsuan Yang
- Yi Yang
- Trevor Darrell
- Xiaogang Wang
- Alan L. Yuille
In Federated & Distributed Learning
- Qiang Yang
- Murali Annavaram
- Lingjuan Lyu
- Chaoyang He
- Praneeth Vepakomma
- Yang Liu
- Medhi Bennis
- Dusist Niyato
In Privacy & Security
- Reza Shokri
- Qiang Yang
- Lingjuan Lyu
- Congzheng Song
- Vitaly Shmatikov
- Mario Fritz
- Ramesh Raskar
- Praneeth Vepakomma
Citations in the corpus
- Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, background): “Recent studies [29, 40, 44] explored privacy risks in federated learning, which are orthogonal to our study.”
- NoPeek: Information leakage reduction to share activations in distributed deep learning (2020, background): “The works in [45, 46, 47] are examples of such schemes for transfer learning, subspace clustering.”
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “…trustworthiness challenge, existing research focuses on developing new adversarial attack and defense techniques to make federated learning robust [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 22], and proposing methods such as differential privacy (DP) and secure multiparty computation (SMPC)…”
- Harnessing Wireless Channels for Scalable and Privacy-Preserving Federated Learning (2020, background): “4 Privacy Analysis Revealing the local model update trajectory is vulnerable to model inversion and reconstruction attacks [13,14].”
- Privacy in Deep Learning: A Survey (2020, methodology): “-Differential Privacy ( -DP).”
- Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “Besides, many empirical studies are investigating the effect of system heterogeneity [31, 55], the power of scalability [22], strategies of preserving privacy [6, 27], and saving communication [9].”
- On the Convergence of FedAvg on Non-IID Data (2019, background): “The research work on FL is focused on the communication-efficiency [10, 16, 21, 27] and data privacy [1, 2, 4, 5, 17].”
- Federated Machine Learning (2019, background): “But it may be subject to attack in another security model by a malicious participant training a Generative Adversarial Network (GAN) in the collaborative learning process [29].”
- Mitigating Sybils in Federated Learning Poisoning (2018, background): “For example, differentially private [15] and securely aggregated [10] additions have been released; however, the use of federated learning in multi-party settings still presents new privacy and security vulnerabilities [19], [3].”
- Exploiting Unintended Feature Leakage in Collaborative Learning (2018, methodology): “The informal property violated by the attacks of [2, 16, 17, 25] is: “a classifier should prevent users from generating an input that belongs to a particular class or even learning what”
- Differential Privacy for Growing Databases (2018, background): “Analysis of sensitive data without formal privacy guarantees has led to numerous privacy violations in practice, for tasks such as recommender systems [CKN + 11], targeted advertising [Kor10], data anonymization [NS08], and deep learning [HAPC17].”
- Personalized and Private Peer-to-Peer Machine Learning (2017, background): “We conclude this section by briefly mentioning the recent work of Hitaj et al. (2017) describing an attack against differentially private collaborative deep learning approaches (such as Shokri and Shmatikov, 2015).”
- Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks (2018, background): “Hitaj et al. [ 25] proposed an attack on collaborative learning to generate average samples of a class.”
- Contamination Attacks and Mitigation in Multi-Party Machine Learning (2018, background): “This setting already encourages each party to supply data that benefits the others as opposed to supplying a dummy dataset with the goal of either learning more information about other parties or decreasing the overall accuracy of the model [4, 19, 38].”
- Biscotti: A Ledger for Private and Secure Peer-to-Peer Machine Learning (2018, background): “Due to the vulnerabilities of secure aggregation, we do not consider information leakage attacks with side information [21, 45] and class-level information leakage attacks on federated learning [24], which attempt to learn the properties of an entire target class.”
- Knock Knock, Who's There? Membership Inference on Aggregate Location Data (2017, background): “[14] present active inference attacks on deep neural networks in collaborative settings, while Hayes et al.”
- Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning (2018, background): “Compared to the GAN-based attack [4], mGAN-AI incorporates discrimination on identity that is also fed to G as an important condition.”