When Machine Unlearning Jeopardizes Privacy

By Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, Yang Zhang. 2020

325 citations recorded by the source corpus.

The right to be forgotten states that a data owner has the right to erase their data from an entity storing it. In the context of machine learning (ML), the right to be forgotten requires an ML model owner to remove the data owner's data from the training set used to build the ML model, a process known asmachine unlearning. While originally designed to protect the privacy of the data owner, we argue that machine unlearning may leave some imprint of the data in the ML model and thus create unintended privacy risks. In this paper, we perform the first study on investigating the unintended information leakage caused by machine unlearning. We propose a novel membership inference attack that leverages the different outputs of an ML model's two versions to infer whether a target sample is part of the training set of the original model but out of the training set of the corresponding unlearned model. Our experiments demonstrate that the proposed membership inference attack achieves strong performance. More importantly, we show that our attack in multiple cases outperforms the classical membership inference attack on the original ML model, which indicates that machine unlearning can have counterproductive effects on privacy. We notice that the privacy degradation is especially significant for well-generalized ML models where classical membership inference does not perform well. We further investigate four mechanisms to mitigate the newly discovered privacy risks and show that releasing the predicted label only, temperature scaling, and differential privacy are effective. We believe that our results can help improve privacy protection in practical implementations of machine unlearning. \footnoteOur code is available at \urlhttps://github.com/MinChen00/UnlearningLeaks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning with Recoverable Forgetting (2022, positive): “As machine unlearning is studied for data privacy purpose, Chen et al. [7] firstly study on investigating the unintended information leakage caused by machine unlearning.”
Unrolling SGD: Understanding Factors Influencing Machine Unlearning (2021, background): “Machine unlearning has since been extended to deep neural networks (DNNs) [7, 8, 9, 10, 11, 12, 13], and its privacy implications have been studied [14, 15, 16].”
Remember What You Want to Forget: Algorithms for Machine Unlearning (2021, background): “Other works in this space focus on exploring privacy risks (Chen et al., 2020) and veriﬁcation (Sommer et al., 2020) in machine unlearning settings.”
Machine Unlearning for Random Forests (2020, background): “However, one must be aware that DaRE models (as well as any unlearning method) can leak which instances are deleted if an adversary has access to the model before and after the deletion (Chen et al., 2020).”
Membership Inference Attacks on Machine Learning: A Survey (2021, unknown): “[21, 26] ChestX-ray8 [206] 108,948 1024 × 1024 × 1 32,717 [118]”
Descent-to-Delete: Gradient-Based Methods for Machine Unlearning (2020, methodology): “Chen et al. (2020) observe that deterministic deletion procedures such as SISA (Bourtoule et al., 2019) can exacerbate privacy problems when an attacker can observe both the model before and after the deletion of a particular user’s data point, and show how to perform membership inference attacks ag”
Zero-Shot Machine Unlearning (2022, unknown): “The possible information leakage in the existing machine unlearning methods as discussed in [23] also motivated us to look towards a completely data-free approach to unlearning.”
A Survey of Machine Unlearning (2022, unknown): “ShallowAttack [23] Python Pytorch Model-agnostic https://github.”
Fast Yet Effective Machine Unlearning (2021, background): “Some researchers have studied the unintended privacy risks resulting from the existing unlearning methods [34], [35].”