Gradient Projection Memory for Continual Learning

By Gobinda Saha, Isha Garg, Kaushik Roy. 2021

459 citations recorded by the source corpus.

The ability to learn continually without forgetting the past tasks is a desired attribute for artificial learning systems. Existing approaches to enable such learning in artificial neural networks usually rely on network growth, importance based weight update or replay of old data from the memory. In contrast, we propose a novel approach where a neural network learns new tasks by taking gradient steps in the orthogonal direction to the gradient subspaces deemed important for the past tasks. We find the bases of these subspaces by analyzing network representations (activations) after learning each task with Singular Value Decomposition (SVD) in a single shot manner and store them in the memory as Gradient Projection Memory (GPM). With qualitative and quantitative analyses, we show that such orthogonal gradient descent induces minimum to no interference with the past tasks, thereby mitigates forgetting. We evaluate our algorithm on diverse image classification datasets with short and long sequences of tasks and report better or on-par performance compared to the state-of-the-art approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Theauxiliaryinformationwithlowstoragecost,suchasclass statistics[79],[140]andattentionmaps[141],[142],canbe further incorporated to maintain old knowledge.”
The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “For continual linear regression, the methods of Zeng et al. (2019); Saha et al. (2021); Wang et al. (2021b); Kong et al. (2022) are of the form (9), and so they all converge “approximately” to global minimizers of (7); these methods differ mainly in how the projection Kt−1K>t−1 is approximated and…”
Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer (2022, methodology): “, [3, 6, 25]) aim to update the model with minimal interference introduced to old tasks.”
Continual Learning by Modeling Intra-Class Variation (2022, neutral): “Similar to our work, gradient-based regularization methods (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018; Saha et al., 2021) try to restrict the update directions in a common space where gradients from old classes and new classes have the largest inner product.”
Class Gradient Projection For Continual Learning (2022, methodology): “Recently, a series of continual learning methods combined with orthogonal projection have been proposed [14, 35].”
Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “To evaluate the effectiveness of ORDER for mitigating forgetting in SETS, we compare to SOTA CL baselines, including Experience replay (ER) [15], A-GEM[14], MER [48], GPM [49] and DEGCL [13].”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “In addition, there exist other kinds of methods like gradient-based methods (e.g., [56, 73, 16, 58, 61, 64]), regularization-based ones (e.g. [32, 74, 2, 1]), and expansion-based ones (e.g. [69, 34, 25]).”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “The results execept Vanilla, CPR, and GPM are from [49].”
Incremental Task Learning with Incremental Rank Updates (2022, background): “It has been a popular continual learning setup even in the very recent literature [7,14,11,36,31,40].”
Improving Task-free Continual Learning by Distributionally Robust Memory Evolution (2022, methodology): “…methods (Lopez-Paz & Ranzato, 2017; Nguyen et al., 2018; Kirkpatrick et al., 2017; Zenke et al., 2017; von Oswald et al., 2019; Wang et al., 2021; Saha et al., 2021; Pham et al., 2021; Wang et al., 2022) are only applicable to the task-aware setting, where there are clear task deﬁnitions and…”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “Saha, Garg, and Roy (2021) proposed an orthogonal method by taking new task’s gradient steps in the orthogonal direction to the gradient subspaces deemed important for the past tasks.”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “These constraints can be directly applied on the weights [39, 79, 1, 8], intermediary features [32, 15, 83, 19, 17], prediction probabilities [47, 59, 5, 6], or on the gradients [51, 9, 21, 62].”
Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference (2021, background): “However, many CL methods still focus on the ofﬂine setup (Kirk-patrick et al., 2017; Rebufﬁ et al., 2017; Saha et al., 2021) instead of the more realistic online setup.”
Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning (2021, methodology): “In addition, we update the replay bufferM with reservoir sampling as in [31], and then use Singular Value Decomposition (SVD) to recalculate M based on the sampling data in the replay buffer after learning one task following GPM [34].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Apart from the above-mentioned schemes, i.e., reg-ularizing the loss function, gradient and learning rate, Kapoor et al. [53] performed regularization from another perspective.”
Natural continual learning: success is a journey, not (just) a destination (2021, background): “Recent projection-based methods (Duncker et al., 2020; Zeng et al., 2019; Saha et al., 2021) have addressed the continual learning problem using an update rule of the form where P L and P R are projection matrices constructed from previous tasks which encourage parameter updates that do not…”
Optimizing Reusable Knowledge for Continual Learning via Metalearning (2021, methodology): “In our experiments, we compare MARK with recent state of the art methods: Hard Attention Tasks (HAT) [7], A-GEM [28], Adversarial Continual Learning (ACL) [18], and GPM [29].”
Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual Learners (2022, methodology): “To overcome catastrophic forgetting in artificial neural networks, three types of methods have been explored in the past few years, replay [15, 3, 6, 20, 10], regularization [9, 19, 8, 4, 17, 13, 5, 7, 16], and expand [22, 14, 1, 21].”