Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting

By Hippolyt Ritter, Aleksandar Botev, David Barber. 2018

384 citations recorded by the source corpus.

We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate the posterior after every task with a Gaussian, leading to a quadratic penalty on changes to the weights. The Laplace approximation requires calculating the Hessian around a mode, which is typically intractable for modern architectures. In order to make our method scalable, we leverage recent block-diagonal Kronecker factored approximations to the curvature. Our algorithm achieves over 90% test accuracy across a sequence of 50 instantiations of the permuted MNIST dataset, substantially outperforming related methods for overcoming catastrophic forgetting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Toward Understanding Catastrophic Forgetting in Continual Learning (2019, methodology): “Recent work attempted to tackle this problem either by better training algorithms [21, 25, 33, 51], structure sharing [36, 41, 49], episodic memory [9, 27, 29], machine-generated pseudo-data [18, 26, 42], or a combination of these approaches [29, 39].”
Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines (2018, background): “This evaluation scenario is widely adopted [3, 4, 5, 6, 7, 8, 9], despite criticism that it is less challenging in terms of forgetting [10].”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “diagonal approximation [179], [227] or a Kronecker-factored approximation [296], [373].”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, methodology): “[42] successfully applied the KFAC approach to online learning scenarios.”
Continual Learning via Sequential Function-Space Variational Inference (2023, methodology): “…2018; Chaudhry et al., 2018; Kirkpatrick et al., 2017; Lee et al., 2017; Liu et al., 2018; Loo et al., 2020; Nguyen et al., 2018; Park et al., 2019; Ritter et al., 2018; Schwarz et al., Proceedings of the 39 th International Conference on Machine Learning , Baltimore, Maryland, USA, PMLR 162, 2022.”
Incorporating neuro-inspired adaptability for continual learning in artificial intelligence (2023, methodology): “Representative strategies include selectively stabilizing parameters [7–11], recovering old data distributions [12–16] and allocating dedicated parameter subspaces [17, 18], etc.”
Learnability and Algorithm for Continual Learning (2023, methodology): “Many existing approaches have used similar approaches (Ritter et al., 2018; Schwarz et al., 2018; Xu & Zhu, 2018; Castro et al., 2018; Dhar et al., 2019; Lee et al., 2019; Ahn et al., 2019; Liu et al., 2020a).”
The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “There are now many existing methods for (deep) continual learning, including regularization-based methods (Kirkpatrick et al., 2017; Zenke et al., 2017; Liu et al., 2018; Ritter et al., 2018; Lesort et al., 2019; Park et al., 2019; Yin et al., 2020; Zhou et al., 2021; Heckel, 2022), memory-based met”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, methodology): “To provide a thorough evaluation, we investigate how the following existing regularization techniques (coupled with replay strategies such as ER-ACE and DER++) compare with our solution: • Stable SGD (sSGD) [54], which introduces some specific alterations to the model’s training regime that bias the”
DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, methodology): “Regularization-based methods aim to penalize the update either in function space [8, 24] or in parameter space [2, 9, 18, 20, 22, 23, 32, 38, 45].”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, background): “Many other papers use related approaches (Zenke, Poole, and Ganguli 2017; Fernando et al. 2017; Aljundi et al. 2017; Ritter, Botev, and Barber 2018; Xu and Zhu 2018; Kemker and Kanan 2018; Parisi et al. 2018; Ahn et al. 2019; Hu et al. 2021; Dhar et al. 2019; Adel, Zhao, and Turner 2020).”
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, background): “…experience replay (Lin, 1992; Chaudhry et al., 2019b; Aljundi et al., 2019a; Riemer et al., 2019; Rolnick et al., 2019; Wu et al., 2019), regularization (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018; Ritter et al., 2018), to fast and slow frameworks (Pham et al., 2020; 2021).”
Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation (2022, background): “regularization-based methods impose constraints on the update of neural weights important to previous tasks to alleviate catastrophic forgetting (Li and Hoiem, 2017; Kirkpatrick et al., 2017; Zenke et al., 2017; Ritter et al., 2018).”
TRGP: Trust Region Gradient Projection for Continual Learning (2022, methodology): “, (Kirkpatrick et al., 2017; Lee et al., 2017; Chaudhry et al., 2018a; Dhar et al., 2019; Ritter et al., 2018; Schwarz et al., 2018; Zenke et al., 2017)) protect the old tasks by adding regularization terms in the loss function to penalize the model change on their important weights.”
Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System (2022, background): “These can be broadly categorized into regularization-based methods (Farajtabar et al., 2020; Kirkpatrick et al., 2017; Ritter et al., 2018; Zenke et al., 2017) which penalizes changes in the network weights, network expansion-based methods (Rusu et al., 2016; Yoon et al., 2017) which dedicate a…”
Continual Learning with Recursive Gradient Optimization (2022, methodology): “…goal of these methods are variants of Lk(θ) +λFRLLk (θ), derived from Bayesian posterior approximation with Gaussian prior (Kirkpatrick et al., 2017; Nguyen et al., 2018) or approximation of the KL-divergence used in natural gradient descent (Amari, 1998; Ritter et al., 2018; Tseran et al., 2018).”
Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning (2021, methodology): “For example, [32] recursively uses a Gaussian Laplace approximation of the Hessian to approximate the posterior after every task, [8] adjusts the learning rate according to the uncertainty defined by the probability distribution of the network weights.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Ritter et al. [32] updated a quadratic penalty term in the loss function for every task by the block-diagonal Kronecker factored approximation of the Hessian matrix for considering the intra-layer parameter interaction simultaneously.”