Continual Learning in Low-rank Orthogonal Subspaces

By Arslan Chaudhry, Naeemullah Khan, Puneet Kumar Dokania, Philip H. S. Torr. 2020

170 citations recorded by the source corpus.

In continual learning (CL), a learner is faced with a sequence of tasks, arriving one after the other, and the goal is to remember all the tasks once the continual learning experience is finished. The prior art in CL uses episodic memory, parameter regularization or extensible network structures to reduce interference among tasks, but in the end, all the approaches learn different tasks in a joint vector space. We believe this invariably leads to interference among different tasks. We propose to learn tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Further, to keep the gradients of different tasks coming from these subspaces orthogonal to each other, we learn isometric mappings by posing network training as an optimization problem over the Stiefel manifold. To the best of our understanding, we report, for the first time, strong results over experience-replay baseline with and without memory on standard classification benchmarks in continual learning. The code is made publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Orthog-Subspace [64] performs continual learning with orthogonal low-rank vector subspaces and keeps the”
Learnability and Algorithm for Continual Learning (2023, methodology): “Orthogonal projection methods learn each task in an orthogonal space to other tasks to reduce task interference or CF (Zeng et al., 2019; Kim & Liu, 2020; Chaudhry et al., 2020; Lin et al., 2022).”
The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “Peng & Risteski (2022) performed one such analysis, based on prior works on orthogonal gradient descent (Farajtabar et al., 2020; Chaudhry et al., 2020) and matrix factorization (Ye & Du, 2021).”
A Theoretical Study on Solving Continual Learning (2022, background): “[51] Arslan Chaudhry, Naeemullah Khan, Puneet K.”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “We use ResNet-18 [11] as the backbone [4,6,10,11].”
Incremental Task Learning with Incremental Rank Updates (2022, methodology): “For missing comparisons, we trained the models using same procedure as outlined in [7].”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “Chaudhry et al. (2020) proposed an orthogonal method by learning different tasks in different orthogonal subspace.”
Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions (2022, methodology): “Due to the quadratic nature of the loss, and the appearance of “cross-terms”, this turns out to require both column and row orthogonality, and interestingly deviates from the practically common OGD method [10, 5].”
Continual Learning with Recursive Gradient Optimization (2022, methodology): “Permuted MNIST (Goodfellow et al., 2014; Kirkpatrick et al., 2017) and Rotated MNIST (Chaudhry et al., 2020) are variants of MNIST dataset of handwritten digits (LeCun, 1998) with 20 tasks applying random permutations of the input pixels and random rotations of the original images respectively.”
Wide Neural Networks Forget Less Catastrophically (2021, background): “…of experience replay with new task (Shin et al., 2017; Kirichenko et al., 2021; Rolnick et al., 2019; Riemer et al., 2018; Chaudhry et al., 2019; Chaudhry et al., 2020; Balaji et al., 2020) or use it as a set of optimization constraints (Lopez-Paz and Ran-zato, 2017; Chaudhry et al., 2018;…”
Kernel Continual Learning (2021, methodology): “The accuracy and forgetting scores in Table 5 for Rotated MNIST, Permuted MNIST and Split CIFAR100 are all adopted from (Mirzadeh et al., 2020), and results for miniImageNet are from (Chaudhry et al., 2020).”
Federated Continual Learning with Weighted Inter-client Transfer (2020, methodology): “We followed metrics for accuracy and forgetting from recent works (Chaudhry et al., 2020; Mirzadeh et al., 2020; 2021).”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “One mechanism for imposing orthogonality is to use precomputed task-specific orthogonal matrices to project the feature space of each task (Chaudhry et al., 2020).”
A Multi-Head Model for Continual Learning via Out-of-Distribution Replay (2022, methodology): “Some such methods (Kim & Liu, 2020; Chaudhry et al., 2020; Saha et al., 2021) require task-id for more complex tasks to recall the task-specific projection at inference.”
Forget-free Continual Learning with Winning Subnetworks (2022, background): “…to tackle the challenges for continual learning, such as catastrophic forgetting: Regularization-based approaches (Kirkpatrick et al., 2017; Chaudhry et al., 2020; Jung et al., 2020; Titsias et al., 2020; Mirzadeh et al., 2021) lessen the catastrophic forgetting by imposing regularization…”
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (2024, methodology): “This notion has been widely used in different domains of machine learning, including meta-learning and continual learning (Lee & Choi, 2018; Chaudhry et al., 2020).”
Effects of Auxiliary Knowledge on Continual Learning (2022, result): “A similar approach was recently taken by [19].”
Formalizing the Generalization-Forgetting Trade-off in Continual Learning (2021, background): “For example, common solutions to the catastrophic forgetting issue include (1) representation-driven approaches [49, 25], (2) regularization approaches [27, 2, 34, 16, 48, 48, 24, 37, 10, 43], and (3) memory/experience replay [31, 32, 11, 12, 17].”
Online Continual Learning through Mutual Information Maximization (2022, methodology): “7 are recent online CL baselines: ER (Chaudhry et al., 2020), AGEM (Chaudhry et al., 2018), MIR (Aljundi et al., 2019a), ASER (Shim et al., 2021), GSS (Aljundi et al., 2019b), GDumb (Prabhu et al., 2020), and SCR (Mai et al., 2021).”
Learning to Predict Gradients for Semi-Supervised Continual Learning (2022, background): “There are a number of works that can be cast into the category of task-incremental learning [5], [7], [8], [12], [14], [15], [25]–[27].”