Understanding the Role of Training Regimes in Continual Learning

By Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, Hassan Ghasemzadeh. 2020

278 citations recorded by the source corpus.

Catastrophic forgetting affects the training of neural networks, limiting their ability to learn multiple tasks sequentially. From the perspective of the well established plasticity-stability dilemma, neural networks tend to be overly plastic, lacking the stability necessary to prevent the forgetting of previous knowledge, which means that as learning progresses, networks tend to forget previously seen tasks. This phenomenon coined in the continual learning literature, has attracted much attention lately, and several families of approaches have been proposed with different degrees of success. However, there has been limited prior work extensively analyzing the impact that different training regimes -- learning rate, batch size, regularization method-- can have on forgetting. In this work, we depart from the typical approach of altering the learning algorithm to improve stability. Instead, we hypothesize that the geometrical properties of the local minima found for each task play an important role in the overall degree of forgetting. In particular, we study the effect of dropout, learning rate decay, and batch size, on forming training regimes that widen the tasks' local minima and consequently, on helping it not to forget catastrophically. Our study provides practical insights to improve stability via simple yet effective techniques that outperform alternative baselines.

Subfields: Artificial Intelligence, Education, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Education

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “The alternatives include using an approximation of the Hessian [81], [85] or calculating δ only along the trajectory of forward and backward parameter changes [86], [87].”
Incorporating neuro-inspired adaptability for continual learning in artificial intelligence (2023, background): “In practice, sequential learning of each task by minθ∈Θ ÊDt(θ) can find multiple solutions but with different generalizability for EDt(θ) and ED1:t−1(θ), where a solution with flatter loss landscape given a similar loss value typically acquires better generalizability and is therefore more robust to”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, methodology): “To provide a thorough evaluation, we investigate how the following existing regularization techniques (coupled with replay strategies such as ER-ACE and DER++) compare with our solution: • Stable SGD (sSGD) [54], which introduces some specific alterations to the model’s training regime that bias the”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “To evaluate the performance, we use two standard metrics: a) Average accuracy (ACC) [26,32] is the average test accuracy evaluated on all tasks after learning all tasks sequentially; b) Backward Transfer (BWT) [7,26] is the average performance decrease of the network on previous tasks after new lear”
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, positive): “Flatness of Loss Landscape provides a conceptual explanation of generalization for deep neural networks, which is recently introduced to understand catastrophic forgetting in continual learning [31,42,11,6,25].”
Architecture Matters in Continual Learning (2022, background): “Mirzadeh et al. (2020) analyzed the BN in continual learning through the generalization lens.”
Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, neutral): “We deepen the discussion of our previous work about the geometry of local minima in CL, thus strengthening what we and other authors [25] have recently stated.”
An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, background): “Mirzadeh et al. (2020b) argues that the geometric properties of the local minima found for each task play an important role in forgetting, and they propose to modify the training regime (learning rate decay, batch size, dropout) to widen the tasks’ local minima.”
Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning (2021, methodology): “The difference between these two setups is that TIL allows using task information during inference ( i.e ., knowing what task does test data belong to) but the CIL does not.”
Wide Neural Networks Forget Less Catastrophically (2021, background): “In the same vein, Mirzadeh et al. (2020b;a) observed that changing the learning regime (i.e., imposing a learning rate decay, decreasing batch size, employing dropout, etc.) also leads to a small change in parameters from the optimum of previous tasks resulting in less forgetting.”
Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference (2021, methodology): “For the LR scheduling, we ﬁrst note that other CL methods use either (1) exponential decay (Rebufﬁ et al., 2017; Kirkpatrick et al., 2017; Mirzadeh et al., 2020) or (2) constant LR.”
Representational Continuity for Unsupervised Continual Learning (2021, methodology): “Next, we evaluate the quality of our pre-trained representations using a K-nearest neighbor (KNN) classiﬁer (Wu et al., 2018) To validate knowledge transfer of the learned representations, we adopt the metrics from the SCL literature (Chaudhry et al., 2019b; Mirzadeh et al., 2020).”
Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning (2021, methodology): “Compared with our visualization, [26] only consider one task and plot the loss landscape along the directions of its Hessian eigenvectors, which only reflects some of the relationship between forgetting and landscape.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Aside from the regularization ofmodel parameters, a few other methods [20, 46] proposed to regularize the learning rate, which have been shown to be effective in mitigating catastrophic forgetting.”
AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021, background): “Actually, there are some concurrent works also associating the ﬂat local minimal with forgetting mitigation in CL from different viewpoints (e.g., in CPR [3] and stable SGD [31]).”
Kernel Continual Learning (2021, methodology): “The accuracy and forgetting scores in Table 5 for Rotated MNIST, Permuted MNIST and Split CIFAR100 are all adopted from (Mirzadeh et al., 2020), and results for miniImageNet are from (Chaudhry et al., 2020).”
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “Similarly, a series of works has started a systematic empirical analysis of this phenomenon in deep networks (Parisi et al., 2019; Mirzadeh et al., 2020; Neyshabur et al., 2020; Nguyen et al., 2019; Ruder & Plank, 2017).”