Orthogonal Gradient Descent for Continual Learning

By Mehrdad Farajtabar, Navid Azizan, Alex Mott, Ang Li. 2019

536 citations recorded by the source corpus.

Neural networks are achieving state of the art and sometimes super-human performance on learning tasks across a variety of domains. Whenever these problems require learning in a continual or sequential manner, however, neural networks suffer from the problem of catastrophic forgetting; they forget how to solve previous tasks after being trained on a new task, despite having the essential capacity to solve both tasks if they were trained on both simultaneously. In this paper, we propose to address this issue from a parameter space perspective and study an approach to restrict the direction of the gradient updates to avoid forgetting previously-learned data. We present the Orthogonal Gradient Descent (OGD) method, which accomplishes this goal by projecting the gradients from new tasks onto a subspace in which the neural network output on previous task does not change and the projected gradient is still in a useful direction for learning the new task. Our approach utilizes the high capacity of a neural network more efficiently and does not require storing the previously learned data that might raise privacy concerns. Experiments on common benchmarks reveal the effectiveness of the proposed OGD method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “OGD [113] stores the old gradient directions and rectifies the current gradient orthogonal to them.”
The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “We first connect ICL† to the orthogonal gradient descent method (OGD) of Farajtabar et al. (2020): Proposition 6 (Informal).”
Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer (2022, background): “, [32, 6, 25, 17]) update the model in the orthogonal direction of old tasks without storing the raw data of old tasks.”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “With memory buffers, regularization-based methods can better utilize the old activation knowledge (Rebuffi et al., 2017; Buzzega et al., 2020) and old gradient information (Lopez-Paz & Ranzato, 2017; Wang et al., 2021; Farajtabar et al., 2020).”
Class Gradient Projection For Continual Learning (2022, background): “Recently, the gradient projection based paradigms [14, 35, 44] have achieved remarkable performance in tackling the catastrophic forgetting problem in CL.”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Methods such as GEM [34], A-GEM [10], OGD [14] and NSCL [50] fall under this category.”
One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squares (2022, positive): “We discussed the connection between the proposed method and orthogonal gradient descent (OGD), a practical algo-rithm in continual learning, as well as the recursive least-squares (RLS), a well-known method from adaptive ﬁltering.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “In addition, there exist other kinds of methods like gradient-based methods (e.g., [56, 73, 16, 58, 61, 64]), regularization-based ones (e.g. [32, 74, 2, 1]), and expansion-based ones (e.g. [69, 34, 25]).”
Incremental Task Learning with Incremental Rank Updates (2022, background): “Incremental task learning is a subcategory of continual learning or lifelong learning approaches aim to address the problem of catastrophic forgetting by adapting the network or training process to learn new tasks without forgetting the previously learned ones [23,16,3,2,4,6,28,29,12].”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, positive): “OGD saves previous tasks’ gradients (Farajtabar et al. 2020).”
Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions (2022, methodology): “Due to the quadratic nature of the loss, and the appearance of “cross-terms”, this turns out to require both column and row orthogonality, and interestingly deviates from the practically common OGD method [10, 5].”
TRGP: Trust Region Gradient Projection for Continual Learning (2022, methodology): “To minimize the interference to old tasks, recently a series of studies (Zeng et al., 2019; Farajtabar et al., 2020; Saha et al., 2021) has been carried out to update the model for the new task in the direction orthogonal to the subspace spanned by inputs of old tasks.”
Architecture Matters in Continual Learning (2022, background): “…et al., 2019; Rolnick et al., 2019; Riemer et al., 2018) or incorporate them as part of the optimization process (e.g., gradient projection) (Farajtabar et al., 2020; Chaudhry et al., 2018; Balaji et al., 2020), or learn a generative model (Shin et al., 2017; Kirichenko et al., 2021) or…”
Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System (2022, background): “These can be broadly categorized into regularization-based methods (Farajtabar et al., 2020; Kirkpatrick et al., 2017; Ritter et al., 2018; Zenke et al., 2017) which penalizes changes in the network weights, network expansion-based methods (Rusu et al., 2016; Yoon et al., 2017) which dedicate a…”
Continual Learning with Recursive Gradient Optimization (2022, background): “Since previous samples cannot be accessed to recalculate the gradient, common gradient-based methods (Li et al., 2019; Azizan et al., 2019; Farajtabar et al., 2019) assume that the joint optimal parameter lies in the neighborhood of the previous optimal parameter and use previously calculated gradie”
An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, background): “, 2020); (3) optimization-based approaches that either maintain a space of gradient directions for previous tasks and projects the gradients of a new task in a direction orthogonal to that space, ensuring less disruption of older tasks (Farajtabar et al., 2020) or modify training regimes by specific”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “This increased diversity has a drawback: the memory overhead is too important (97M v.s. 10M parameters).”
Wide Neural Networks Forget Less Catastrophically (2021, background): “Earlier works in continual learning (Farajtabar et al., 2020; Chaudhry et al., 2018; Lopez-Paz and Ranzato, 2017; Riemer et al., 2018) have focused on explicitly forcing the gradients of the new tasks to be orthogonal to those of the previous ones to reduce forgetting among the tasks.”
Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning (2021, background): “Thus, forgetness can be effectively mitigated by training a single network for new tasks by considering diverse information stored in the memory, including the original training samples of old tasks [31, 5, 13], the gradients induced from old tasks [9] and the feature subspace representing old tasks”
Model Zoo: A Growing Brain That Learns Continually (2021, background): “…been the focus of a number of continual learning techniques, e.g., episodic memory-based ones (Lopez-Paz and Ranzato, 2017; Chaudhry et al., 2019a; Farajtabar et al., 2020; Guo et al., 2020a), data replay (Robins, 1995; Shin et al., 2017; Lee et al., 2017), new architectures (Serra et al., 2018),…”