Inverting Gradients - How easy is it to break privacy in federated learning?

By Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, Michael Moeller. 2020

1,706 citations recorded by the source corpus.

The idea of federated learning is to collaboratively train a neural network on a server. Each user receives the current weights of the network and in turns sends parameter updates (gradients) based on local data. This protocol has been designed not only to train neural networks data-efficiently, but also to provide privacy benefits for users, as their input data remains on device and only parameter gradients are shared. But how secure is sharing parameter gradients? Previous attacks have provided a false sense of security, by succeeding only in contrived settings - even for a single image. However, by exploiting a magnitude-invariant loss along with optimization strategies based on adversarial attacks, we show that is is actually possible to faithfully reconstruct images at high resolution from the knowledge of their parameter gradients, and demonstrate that such a break of privacy is possible even for trained deep networks. We analyze the effects of architecture as well as parameters on the difficulty of reconstructing an input image and prove that any input to a fully connected layer can be reconstructed analytically independent of the remaining architecture. Finally we discuss settings encountered in practice and show that even averaging gradients over several iterations or several images does not protect the user's privacy in federated learning applications in computer vision.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Privacy Amplification by Decentralization (2020, background): “In fact, FL provides an additional attack surface as the participants share intermediate updates (Nasr et al., 2019; Geiping et al., 2020).”
Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning (2020, background): “In contrast, conventional distributed training setups that do not employ secure aggregation may reveal extensive information about the private datasets of the users, which has been recently shown in [5]–[7].”
Evaluating Gradient Inversion Attacks and Defenses in Federated Learning (2021, methodology): “Based on this, [Geiping et al., 2020] assumes the attacker knows private labels (see remark at the end of Section 4 in their paper).”
FLOP: Federated Learning on Medical Datasets using Partial Networks (2021, background): “By contrast, FLOP avoids Deep Leakage [9, 38, 41] and protects patients’ privacy by only allowing the server access to a partial modelMu s from each client u.”
Federated Learning with Buffered Asynchronous Aggregation (2021, methodology): “The results for Sent140 are in Figure 6.”
FedScale: Benchmarking Model and System Performance of Federated Learning at Scale (2021, background): “Instead of training a single global model, some efforts resort to training a mixture of models [19, 22], clustering clients over training [27], or enforcing guided client selection [35]; To tackle the scarce and heterogeneous device resource, FedAvg [41] reduces communication cost by performing mult”
Mind the Pad - CNNs can Develop Blind Spots (2020, background): “In contrast, circular padding was shown to limit this ability [6] and to boost shift invariance [31].”
A Framework for Evaluating Gradient Leakage Attacks in Federated Learning (2020, methodology): “Note that CPL using L2 distance offers slightly higher ASR compared to [10] but at much lower attack cost in terms of attack iterations required.”
Federated learning for medical image analysis: A survey (2023, background): “But further research reveals that FL still faces privacy and security risks, including privacy leakage (Geiping et al., 2020; Yin et al., 2021a; Zhu et al., 2019) and poisoning attacks (Lyu”
Privacy-Enhanced Federated Learning Against Poisoning Adversaries (2021, background): “As prior works [4], [5] have shown, an adversary may recover users’ sensitive information such as training samples or memberships by inferring the shared gradients.”
End-to-end privacy preserving deep learning on multi-institutional medical imaging (2021, methodology): “DP model training is able to offer objective privacy guarantees and resilience against model inversion attacks 30,32 .”
Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challenges (2022, methodology): “[64] of an image with label 0 from EMNIST dataset, without any defence, with Gaussian noise defence and with Gradient compresion defence.”
FedScale: Benchmarking Model and System Performance of Federated Learning (2021, background): “Instead of training a single global model, some efforts resort to training a mixture of models [10, 12], clustering clients over training [16], or enforcing guided client selection [24]; To tackle the scarce and heterogeneous device resource, FedAvg [29] reduces communication cost by performing mult”
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses (2020, background): “rely on direct access to model updates, which can reveal information about user data [202], [203].”
PPFL: privacy-preserving federated learning with trusted execution environments (2021, methodology): “Privacy risks depend on the aggregation level and frequency with which they happen, when exposing the model or its layers [16, 45, 47].”
Privacy preservation in federated learning: An insightful survey from the GDPR perspective (2020, methodology): “, all class members depict this person) using MI by deriving a correct weighted probability estimation for the target feature vectors [112, 43].”