Machine Unlearning
By Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, Nicolas Papernot. 2019
1,538 citations recorded by the source corpus.
Once users have shared their data online, it is generally difficult for them to revoke access and ask for the data to be deleted. Machine learning (ML) exacerbates this problem because any model trained with said data may have memorized it, putting users at risk of a successful privacy attack exposing their information. Yet, having models unlearn is notoriously difficult.We introduce SISA training, a framework that expedites the unlearning process by strategically limiting the influence of a data point in the training procedure. While our framework is applicable to any learning algorithm, it is designed to achieve the largest improvements for stateful algorithms like stochastic gradient descent for deep neural networks. SISA training reduces the computational overhead associated with unlearning, even in the worst-case setting where unlearning requests are made uniformly across the training set. In some cases, the service provider may have a prior on the distribution of unlearning requests that will be issued by users. We may take this prior into account to partition and order data accordingly, and further decrease overhead from unlearning.Our evaluation spans several datasets from different domains, with corresponding motivations for unlearning. Under no distributional assumptions, for simple learning tasks, we observe that SISA training improves time to unlearn points from the Purchase dataset by 4.63×, and 2.45× for the SVHN dataset, over retraining from scratch. SISA training also provides a speed-up of 1.36× in retraining for complex learning tasks such as ImageNet classification; aided by transfer learning, this results in a small degradation in accuracy. Our work contributes to practical data governance in machine unlearning.
Subfields: Artificial Intelligence, Management Science and Operations Research, Privacy & Security
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Luke S. Zettlemoyer
- Mohit Bansal
- Sergey Levine
- Jianfeng Gao
- Yoshua Bengio
- Quoc V. Le
- Ruslan Salakhutdinov
In Management Science and Operations Research
- Enrique Herrera-Viedma
- Francisco Herrera
- Zeshui Xu
- Francisco Chiclana
- Luis Martínez-López
- Seth Neel
- Aaron Roth
- Luke S. Zettlemoyer
In Privacy & Security
- Philip S. Yu
- Nicolas Papernot
- Lichao Sun
- Ananda Theertha Suresh
- Jayadev Acharya
- Tianhao Wang
- Shruti Tople
- Seth Neel
Citations in the corpus
- Foundation Models and Fair Use (2023, background): “And in other cases, one might seek to have a post-hoc mechanism to delete information about a particular training example from a model Bourtoule et al. (2021)—such as if a DMCA request for takedown is provided.”
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Many efforts in this direction are closely related to continuallearning,suchaslearningseparatemodelswithsubsets oftrainingsamples[307],utilizinghistoricalparametersand gradients[308],removingoldknowledgefromparameterswith Fisherinformationmatrix[309],addingadaptiveparameterson…”
- Learning with Recoverable Forgetting (2022, neutral): “The concept of unlearning is firstly introduced by Bourtoule et al. [2] to eliminate the effect of data point(s) on the already trained model.”
- Learning with Selective Forgetting (2021, background): “General approaches are to train multiple small models on separated subsets of the training data to prevent re-training the whole model [Bourtoule et al. , 2019] or to utilize vestiges of the learning process, i.e., the stored learned model parameters and their gradients [Wu et al. , 2020].”
- Mixed-Privacy Forgetting in Deep Networks (2020, methodology): “Recently, the problem of forgetting has received considerable attention [15, 16, 13, 19, 5, 24, 35, 41, 6, 39, 12, 7, 36], but solutions have focused on simpler machine learning problems such as linear logistic regression.”
- DeltaGrad: Rapid retraining of machine learning models (2020, background): “(Bourtoule et al., 2019) propose the SISA (or Sharded, Isolated, Sliced, Aggregated) training framework for “un-learning”, which relies on ideas similar to distributed training.”
- A Survey on Negative Transfer (2020, background): “1) Machine Unlearning: Aims to preserve data privacy in trained models by unlearning sensitive samples [33], [34].”
- Amnesiac Machine Learning (2020, methodology): “Recently, Bourtoule et al. (Bourtoule et al. 2019) introduced a method for dealing with individual data removal requests.”
- Detecting Pretraining Data from Large Language Models (2023, background): “…has emerged as a solution for purging data from machine learning models, and various machine unlearning methods have been introduced (Ginart et al., 2019; Liu et al., 2020; Wu et al., 2020; Bourtoule et al., 2021; Izzo et al., 2021; Sekhari et al., 2021; Gupta et al., 2021; Ye et al., 2022).”
- A Survey of Machine Unlearning (2022, neutral): “) is the parameterized mapping function from the input space Z to the output space (i.e., the machine weak unlearning [4].”
- A Unified and General Framework for Continual Learning (2024, background): “Machine Unlearning (Guo et al., 2020; Wu et al., 2020; Bourtoule et al., 2021; Ullah et al., 2021) refers to the process of removing or erasing previously learned information or knowledge from a pre-trained model to comply with privacy regulations (Ginart et al., 2019).”
- Analyzing Information Leakage of Updates to Natural Language Models (2019, methodology): “[5] propose an exact deletion methodology that aggregates models trained on disjoint data shards, trading storage for computation such that only shards that contain deleted points need to be retrained.”
- Machine Unlearning of Features and Labels (2021, methodology): “[4] address this problem and propose a strategy for unlearning data instances from general classification models.”
- Towards Unbounded Machine Unlearning (2023, background): “[Bourtoule et al., 2021] proposed a training framework by sharding data and creating multiple models, enabling exact unlearning of certain data partitions.”
- TOFU: A Task of Fictitious Unlearning for LLMs (2024, neutral): “With unlimited resources, one could test the ( ε , δ ) -unlearning condition of indistinguishability (Bourtoule et al., 2021; Sekhari et al., 2021) by training many models and performing hypothesis tests—and this is done in practice when feasible (Carlini et al., 2022; Pawelczyk et al., 2023).”
- Variational Bayesian Unlearning (2020, background): “Since existing unlearning methods often use their model predictions to construct their loss functions [3, 4, 12, 14], we have initially considered doing likewise (albeit in the Bayesian context) by defining the loss function as the KL divergence between the approximate predictive distribution qu(y|D”