Variational Bayesian Unlearning

By Quoc Phong Nguyen, Bryan Low, Patrick Jaillet. 2020

191 citations recorded by the source corpus.

This paper studies the problem of approximately unlearning a Bayesian model from a small subset of the training data to be erased. We frame this problem as one of minimizing the Kullback-Leibler divergence between the approximate posterior belief of model parameters after directly unlearning from erased data vs. the exact posterior belief from retraining with remaining data. Using the variational inference (VI) framework, we show that it is equivalent to minimizing an evidence upper bound which trades off between fully unlearning from erased data vs. not entirely forgetting the posterior belief given the full data (i.e., including the remaining data); the latter prevents catastrophic unlearning that can render the model useless. In model training with VI, only an approximate (instead of exact) posterior belief given the full data can be obtained, which makes unlearning even more challenging. We propose two novel tricks to tackle this challenge. We empirically demonstrate our unlearning methods on Bayesian models such as sparse Gaussian process and logistic regression using synthetic and real-world datasets.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Learning with Recoverable Forgetting (2022, positive): “Nguyen et al. [33] study the problem of approximately unlearning a Bayesian model from a small subset of the training data to be erased.”
Learning with Selective Forgetting (2021, methodology): “Variational Bayesian inference also provides a compelling approach for MU [Nguyen et al. , 2020].”
Mixed-Privacy Forgetting in Deep Networks (2020, methodology): “[36] proposed an unlearning mechanism for logistic regression and gaussian processes in a Bayesian setting using variational inference.”
Measuring Forgetting of Memorized Training Examples (2022, background): “Algorithms for machine unlearning have been proposed for k -means [26], empirical risk minimization [31, 37, 59, 77, 68], and deep learning [20, 28, 27, 60, 5].”
A Survey of Machine Unlearning (2022, unknown): “wrong output ( ZRF = 0) or random output ( ZRF = ( )”
Remember What You Want to Forget: Algorithms for Machine Unlearning (2021, background): “Finally, there has also been recent empirical and theoretical work in developing deﬁnitions and algorithms for machine unlearning with deep neural networks for application domains in computer vision (Du et al., 2019; Golatkar et al., 2020b,a; Nguyen et al., 2020).”
Ablating Concepts in Text-to-Image Diffusion Models (2023, background): “Another line of work in machine unlearning [9, 21, 23, 22, 42, 8, 67, 60] explores data deletion at user’s request after model training.”
Fast Yet Effective Machine Unlearning (2021, methodology): “Unlearning in a Bayesian setting using variational inference is explored for regression and Gaussian processes in [35].”