SOLA: Continual Learning with Second-Order Loss Approximation

By Dong Yin, Mehrdad Farajtabar, Ang Li. 2020

20 citations recorded by the source corpus.

Neural networks have achieved remarkable success in many cognitive tasks. However, when they are trained sequentially on multiple tasks without access to old data, it is observed that their performance on old tasks tend to drop significantly after the model is trained on new tasks. Continual learning aims to tackle this problem often referred to as catastrophic forgetting and to ensure sequential learning capability. We study continual learning from the perspective of loss landscapes and propose to construct a second-order Taylor approximation of the loss functions in previous tasks. Our proposed method does not require any memorization of raw data or their gradients, and therefore, offers better privacy protection. We theoretically analyze our algorithm from an optimization viewpoint and provide a sufficient and worst-case necessary condition for the gradient updates on the approximate loss function to be descent directions for the true loss function. Experiments on multiple continual learning benchmarks suggest that our method is effective in avoiding catastrophic forgetting and in many scenarios, outperforms several baseline algorithms that do not explicitly store the data samples.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

Continual Learning via Sequential Function-Space Variational Inference (2023, methodology): “Online structured Laplace (Ritter et al., 2018) and second-order loss approximation (Yin et al., 2020a) respectively use Kronecker-factored and low-rank Hessians.”
Continual Learning with Recursive Gradient Optimization (2022, background): “…to lie in a close vicinity of previous solutions by introducing quadratic penalty term to the loss function (Kirkpatrick et al., 2017; Zenke et al., 2017; Yin et al., 2020) or constraining the direction of parameter update (Farajtabar et al., 2019; Chaudhry et al., 2019a; Saha et al., 2021).”
Generalized Variational Continual Learning (2020, methodology): “Equation 4 is an instance of the objective function used by a number of continual learning methods, most notably Online EWC 3 (Kirkpatrick et al., 2017; Schwarz et al., 2018), Online-Structured Laplace (Ritter et al., 2018), and SOLA (Yin et al., 2020).”
Linear Mode Connectivity in Multitask and Continual Learning (2020, methodology): “For example, some regularization methods take a quadratic approximation of the loss of previous tasks (e.g. Kirkpatrick et al., 2017; Yin et al., 2020).”
The Effectiveness of Memory Replay in Large Scale Continual Learning (2020, neutral): “(Kirkpatrick et al., 2017; Zenke et al., 2017; Ritter et al., 2018; Fara-jtabar et al., 2020; Yin et al., 2020; Adel et al., 2020).”
Unifying Importance Based Regularisation Methods for Continual Learning (2022, methodology): “…approaches, including combinations of them [Chaudhry et al., 2018], reﬁnements [Huszár, 2018, Ritter et al., 2018, Chaudhry et al., 2018, Yin et al., 2020], extensions [Schwarz et al., 2018, Liu et al., 2018, Park et al., 2019, Lee et al., 2020] and applications in diﬀerent continual…”