DeltaGrad: Rapid retraining of machine learning models

By Yinjun Wu, Edgar Dobriban, Susan B. Davidson. 2020

275 citations recorded by the source corpus.

Machine learning models are not static and may need to be retrained on slightly changed datasets, for instance, with the addition or deletion of a set of data points. This has many applications, including privacy, robustness, bias reduction, and uncertainty quantifcation. However, it is expensive to retrain models from scratch. To address this problem, we propose the DeltaGrad algorithm for rapid retraining machine learning models based on information cached during the training phase. We provide both theoretical and empirical support for the effectiveness of DeltaGrad, and show that it compares favorably to the state of the art.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “…are closely related to continuallearning,suchaslearningseparatemodelswithsubsets oftrainingsamples[307],utilizinghistoricalparametersand gradients[308],removingoldknowledgefromparameterswith Fisherinformationmatrix[309],addingadaptiveparameterson apre-trainedmodel[310],etc.…”
Learning with Selective Forgetting (2021, positive): “General approaches are to train multiple small models on separated subsets of the training data to prevent re-training the whole model [Bourtoule et al. , 2019] or to utilize vestiges of the learning process, i.e., the stored learned model parameters and their gradients [Wu et al. , 2020].”
Mixed-Privacy Forgetting in Deep Networks (2020, methodology): “Can we forget the cohort of interest without interfering with information about the other data and preserving, to the extent possible, the accuracy of the trained model? Recently, the problem of forgetting has received considerable attention [15, 16, 13, 19, 5, 24, 35, 41, 6, 39, 12, 7, 36], but sol”
Machine Unlearning for Random Forests (2020, background): “, 2020; Schelter, 2020), and deep neural networks (Baumhauer et al., 2020; Golatkar et al., 2020b; Wu et al., 2020); however, there is very limited work addressing the problem of efficient data deletion for tree-based models (Schelter et al.”
DART: Data Addition and Removal Trees (2020, unknown): “…(Guo et al. 2020; Schelter 2020), and deep neural networks (Baumhauer, Sch ¨ ottle, and Zeppelza-uer 2020; Golatkar, Achille, and Soatto 2020b; Wu, Dobriban, and Davidson 2020) (for a comprehensive list of related work, see Section 6); however, there is currently no work addressing the…”
Fast Yet Effective Machine Unlearning (2021, methodology): “Some methods adopt strategic grouping of data in the training procedure and thus enable smooth unlearning by limiting the influence of data points on model learning [4, 46].”
Detecting Pretraining Data from Large Language Models (2023, background): “…has emerged as a solution for purging data from machine learning models, and various machine unlearning methods have been introduced (Ginart et al., 2019; Liu et al., 2020; Wu et al., 2020; Bourtoule et al., 2021; Izzo et al., 2021; Sekhari et al., 2021; Gupta et al., 2021; Ye et al., 2022).”
A Survey of Machine Unlearning (2022, unknown): “15, we only know that the density function of 𝑃𝑟 ( 𝑤 | 𝐷 ) is proportional to a function 𝑓 ( 𝑤 ) = 𝑃𝑟 ( 𝐷 | 𝑤 ) 𝑃𝑟 ( 𝑤 ) , which means 𝑃𝑟 ( 𝑤 | 𝐷 ) cannot be directly sampled.”
Unrolling SGD: Understanding Factors Influencing Machine Unlearning (2021, result): “Note the size and variety of the datasets and models used in our evaluation is comparable or exceeds previous work in the topic of unlearning [22], [9], [11].”
A Unified and General Framework for Continual Learning (2024, background): “Machine Unlearning (Guo et al., 2020; Wu et al., 2020; Bourtoule et al., 2021; Ullah et al., 2021) refers to the process of removing or erasing previously learned information or knowledge from a pre-trained model to comply with privacy regulations (Ginart et al., 2019).”
Continual Learning and Private Unlearning (2022, background): “…this end, typical MU approaches consider training multiple models on different shards of data so that unlearning only requires retraining a speciﬁc model on part of the dataset (Bourtoule et al., 2019), or storing learned model parameters and their gradients for rapid retraining (Wu et al., 2020b).”