Exploiting Unintended Feature Leakage in Collaborative Learning
By Luca Melis, Congzheng Song, Emiliano De Cristofaro, Vitaly Shmatikov. 2018
1,798 citations recorded by the source corpus.
Collaborative machine learning and related techniques such as federated learning allow multiple participants, each with his own training dataset, to build a joint model by training locally and periodically exchanging model updates. We demonstrate that these updates leak unintended information about participants' training data and develop passive and active inference attacks to exploit this leakage. First, we show that an adversarial participant can infer the presence of exact data points -- for example, specific locations -- in others' training data (i.e., membership inference). Then, we show how this adversary can infer properties that hold only for a subset of the training data and are independent of the properties that the joint model aims to capture. For example, he can infer when a specific person first appears in the photos used to train a binary gender classifier. We evaluate our attacks on a variety of tasks, datasets, and learning configurations, analyze their limitations, and discuss possible defenses.
Subfields: Artificial Intelligence, Health Informatics, Federated & Distributed Learning, Privacy & Security
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Health Informatics
- Lingjuan Lyu
- Qiang Yang
- Congzheng Song
- Amir Houmansadr
- Reza Shokri
- Murali Annavaram
- Hamed Haddadi
- H. Brendan McMahan
In Federated & Distributed Learning
In Privacy & Security
- Peter Kairouz
- Abhradeep Thakurta
- Qiang Yang
- Om Thakkar
- H. Brendan McMahan
- Lingjuan Lyu
- Badih Ghazi
- Vitaly Shmatikov
Citations in the corpus
- Advances and Open Problems in Federated Learning (2019, background): “For classic (non-federated) models of computation, understanding a model’s susceptibility to attacks is an active and challenging research area [167, 357, 91, 293].”
- How To Backdoor Federated Learning (2018, background): “Secure aggregation of model updates [7] is essential for privacy because model updates leak sensitive information about participants’ training data [45, 48].”
- Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, background): “Specifically, an attacker could compromise the confidentiality of users’ private training or testing data via various attacks such as model inversion attacks [21, 22], membership inference attacks [40, 49, 53], and property inference attacks [1, 24].”
- The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation (2021, background): “Distributed DP While SecAgg prevents the server from inspecting individual client updates, the server is still able to learn the sum of the updates, which itself may leak potentially sensitive information (Melis et al., 2019; Carlini et al., 2019; Song & Shmatikov, 2019a; Dwork et al., 2015; Song & ”
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “…robust [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 22], and proposing methods such as differential privacy (DP) and secure multiparty computation (SMPC) to protect privacy [25, 26, 27, 28, 29, 30, 31, 32, 33] (see Table 8 in the appendix for a taxonomy of the federated learning literature).”
- IBM Federated Learning: an Enterprise Framework White Paper V0.1 (2020, neutral): “For example, while membership inference attacks [20], where an adversary may try to infer the training data used during the training process by querying the final model, are possible in both centralized and federated learning settings, additional measures need to be considered in federated learning…”
- Salvaging Federated Learning by Local Adaptation (2020, background): “Participants’ model updates leak their training data (Melis et al., 2019), and malicious participants can inject unwanted behaviors into the model (Bagdasaryan et al., 2018; Bhagoji et al., 2019).”
- Secure Federated Submodel Learning (2019, background): “In addition to the fact that the real index reveals a client’s private data, the client’s individual submodel update can still memorize or even allow reconstruction of her private data, namely “model inversion” attack [1], [42]–[45].”
- Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “Besides, many empirical studies are investigating the effect of system heterogeneity [31, 55], the power of scalability [22], strategies of preserving privacy [6, 27], and saving communication [9].”
- On the Convergence of FedAvg on Non-IID Data (2019, neutral): “, 2017) and data privacy (Bagdasaryan et al., 2018; Bonawitz et al., 2017; Geyer et al., 2017; Hitaj et al., 2017; Melis et al., 2019).”
- Differentially Private Learning with Adaptive Clipping (2019, neutral): “However, recent work has shown that it is possible to extract detailed information about individual training examples using only the parameters of a trained model (Fredrikson et al., 2015; Wu et al., 2016; Shokri et al., 2017; Carlini et al., 2018; Melis et al., 2018).”
- Federated Learning: Challenges, Methods, and Future Directions (2019, background): “the need to keep raw data on each device local in federated settings. However, sharing other information such as model updates as part of the training process can also leak sensitive user information [8, 17, 39, 78]. For instance, Carlini et al. [17] demonstrate that one can extract sensitive text p”
- Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks (2018, background): “Concurrently with this work, Melis et al. [27] developed a new set of attacks for the collaborative learning.”
- Federated Model Distillation with Noise-Free Differential Privacy (2020, background): “Nevertheless, recent works have demonstrated that FL may not always provide sufficient privacy guarantees, as communicating model updates throughout the training process can nonetheless reveal sensitive information [Bhowmick et al., 2018; Melis et al., 2019].”
- Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer (2019, methodology): “Privacy: Sharing the model parameters facilitates strong white-box inference attacks, as it opens up the model to the adversary [ 36], [40].”
- Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning (2018, background): “[19] developed a new set of membership inference attacks for the collaborative learning.”