Deep Leakage from Gradients
By Ligeng Zhu, Zhijian Liu, Song Han. 2019
3,062 citations recorded by the source corpus.
Exchanging gradients is a widely used method in modern multi-node machine learning system (e.g., distributed training, collaborative learning). For a long time, people believed that gradients are safe to share: i.e., the training data will not be leaked by gradient exchange. However, we show that it is possible to obtain the private training data from the publicly shared gradients. We name this leakage as Deep Leakage from Gradient and empirically validate the effectiveness on both computer vision and natural language processing tasks. Experimental results show that our attack is much stronger than previous approaches: the recovery is pixel-wise accurate for images and token-wise matching for texts. We want to raise people's awareness to rethink the gradient's safety. Finally, we discuss several possible strategies to prevent such deep leakage. The most effective defense method is gradient pruning.
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Privacy & Security
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Safety, Robustness & Interpretability
- Bo Li
- Xingjun Ma
- Tom Goldstein
- Aleksander Madry
- Dawn Song
- Dimitris Tsipras
- Nicholas Carlini
- Matthias Hein
In Privacy & Security
- Qiang Yang
- Yang Liu
- Chaoyang He
- Murali Annavaram
- Basak Guler
- Yan Kang
- Amir Salman Avestimehr
- Philip S. Yu
Citations in the corpus
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “…challenge, existing research focuses on developing new adversarial attack and defense techniques to make federated learning robust [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 22], and proposing methods such as differential privacy (DP) and secure multiparty computation (SMPC) to…”
- Backdoor attacks and defenses in feature-partitioned collaborative learning (2020, background): “Recent works (Zhu et al., 2019) have shown that gradient sparsification is an effective approach to defense attacks in federated learning.”
- Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning (2020, background): “In contrast, conventional distributed training setups that do not employ secure aggregation may reveal extensive information about the private datasets of the users, which has been recently shown in [5]–[7].”
- Federated Learning from Pre-Trained Models: A Contrastive Learning Approach (2022, background): “[66] Ligeng Zhu, Zhijian Liu, and Song Han.”
- Vertical Federated Learning: Concepts, Advances, and Challenges (2022, methodology): “Adding noise [26], [101], [123], [124] is a basic defense method for reducing leakage in FL. Noise following Laplace distribution or Gaussian distribution is commonly used.”
- Federated Learning with Buffered Asynchronous Aggregation (2021, methodology): “SyncFL methods are also better suited for privacy – training and aggregating updates over a large number of clients render most inference attacks ineffectual (Melis et al., 2019; Zhu and Han, 2020; Geiping et al., 2020; Lam et al., 2021).”
- End-to-end privacy preserving deep learning on multi-institutional medical imaging (2021, methodology): “…that attempt to expose sensitive data, we employ the Improved Deep Leakage from Gradients, iDLG, method with modifications as proposed previously 32 , itself a variant of previously shownn techniques 31,62 . iDLG was found highly successful against the ResNet18 architecture used in our case study.”
- DarKnight: A Data Privacy Scheme for Training and Inference of Deep Neural Networks (2020, methodology): “But as we described in Equation 4, we only provide a single model update computed across all the inputs in a batch, which is similar to the state of art secure aggregation mechanisms used to bound such leakage (Bonawitz et al., 2017; Zhu et al., 2019).”
- Distributed Learning in Wireless Networks: Recent Progress and Future Challenges (2021, background): “Second, since one can use gradient vectors to recover the raw data, called gradient leakage [84], it is necessary to design new compres-”
- See through Gradients: Image Batch Recovery via GradInversion (2021, methodology): “[55] and (ii) federated gradient inversion by Geiping et al.”
- Security and Privacy Challenges of Large Language Models: A Survey (2024, positive): “These aforementioned attack models primarily targeted the language models [106], [28], but they can also be potentially applied to LLMs.”
- FedALA: Adaptive Local Aggregation for Personalized Federated Learning (2022, methodology): “Besides, downloading client models among clients causes high communication overhead in each iteration and also has privacy concerns since the data from other clients can be recovered through these client models (Zhu, Liu, and Han 2019).”
- LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets (2020, background): “However, some works [23, 24] have shown that it is feasible to recover the private training data by eavesdropping the data communicated between the clients and the server.”
- Asynchronous Federated Learning on Heterogeneous Devices: A Survey (2021, background): “Furthermore, the training data is protected securely against gradients leakage attacks [85].”
- A Communication Efficient Collaborative Learning Framework for Distributed Features (2019, background): “Whereas previous research studied data leakage from exposing complete set of model parameters or gradients [24–26], in our protocol model parameters are kept private, and only the intermediate results (such as inner product of model parameters and feature) are exposed.”