Continual Learning by Asymmetric Loss Approximation With Single-Side Overestimation

By Dongmin Park, Seokil Hong, Bohyung Han, Kyoung Mu Lee. 2019

48 citations recorded by the source corpus.

Catastrophic forgetting is a critical challenge in training deep neural networks. Although continual learning has been investigated as a countermeasure to the problem, it often suffers from the requirements of additional network components and the limited scalability to a large number of tasks. We propose a novel approach to continual learning by approximating a true loss function using an asymmetric quadratic function with one of its sides overestimated. Our algorithm is motivated by the empirical observation that the network parameter updates affect the target loss functions asymmetrically. In the proposed continual learning framework, we estimate an asymmetric loss function for the tasks considered in the past through a proper overestimation of its unobserved sides in training new tasks, while deriving the accurate model parameter for the observable sides. In contrast to existing approaches, our method is free from the side effects and achieves the state-of-the-art accuracy that is even close to the upper-bound performance on several challenging benchmark datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Geophysics, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Observing the asymmetric effect of parameter changes on old tasks, ALASSO [110] designs an asymmetric quadratic penalty with one of its sides overestimated.”
Continual Learning via Sequential Function-Space Variational Inference (2023, background): “…regularization includes various modifications to VCL (Ahn et al., 2019; Kessler et al., 2019), uncertainty-guided continual learning in Bayesian neural networks (Ebrahimi et al., 2020), and a variation of SI known as asymmetric loss approximation with single-side overestimation (Park et al., 2019).”
The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “There are now many existing methods for (deep) continual learning, including regularization-based methods (Kirkpatrick et al., 2017; Zenke et al., 2017; Liu et al., 2018; Ritter et al., 2018; Lesort et al., 2019; Park et al., 2019; Yin et al., 2020; Zhou et al., 2021; Heckel, 2022), memory-based met”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, neutral): “Regularization-based methods can be divided into two classes: one is to distill knowledge from the previous model which is trained on the previous tasks [14, 15, 23, 36, 52, 59]; another is to explicitly use a regularizer to penalize the update of important parameters of previous tasks, preventing t”
Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “Some methods [43,48,56,66,67,80,95] play regularization on model parameters to improve its stability.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[29] pointed out that SI [18] may underestimate the loss since it assumes the loss functions are symmetric which is often incorrect.”
Layerwise Optimization by Gradient Decomposition for Continual Learning (2021, positive): “The former approaches, such as EWC [22], SI [55], MAS [1] and ALASSO [35] relied on different estimations of parameters importance, which was usually conducted by exhaustive search [42] or variational Bayesian methods [33, 56].”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “Asymmetric Loss Approximation with Single-Side Overestimation (ALASSO) (Park et al., 2019) can be seen as a direct extension to SI and aims to mitigate its limitations by introducing an asymmetric loss approximation that is motivated from empirical observations.”
SOLA: Continual Learning with Second-Order Loss Approximation (2020, methodology): “ a generative model of their data. The regularization based approaches impose limiting constraints on the weight updates of the neural network according to some relevance score for previous knowledge [21, 33, 42, 39, 29, 47, 35]. These methods provide a better privacy guarantee as they do not explic”
Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpoint. (2020, background): “A large fraction of regularization-based algorithms construct a quadratic regularization function at the end of each task and combine it with the loss of future tasks (Kirkpatrick et al., 2017; Zenke et al., 2017; Ritter et al., 2018; Aljundi et al., 2018; Park et al., 2019).”
Looking Back on Learned Experiences For Class/task Incremental Learning (2022, methodology): “, 2018), and ALASSO (Park et al., 2019) the gap does not seem to be bridged, indicating that the old parameter set’s regu-”