Training Networks in Null Space of Feature Covariance for Continual Learning

By Shipeng Wang, Xiaorong Li, Jian Sun, Zongben Xu. 2021

225 citations recorded by the source corpus.

In the setting of continual learning, a network is trained on a sequence of tasks, and suffers from catastrophic forgetting. To balance plasticity and stability of network in continual learning, in this paper, we propose a novel network training algorithm Adam-NSCL which sequentially optimizes network parameters in the null space of all previous tasks. We first propose two mathematical conditions respectively for achieving network stability and plasticity in continual learning. Based on them, the network training for sequential tasks without forgetting can be simply achieved by projecting the candidate parameter update into the approximate null space of all previous tasks in the network training process, where the candidate parameter update can be generated by Adam. The approximate null space can be derived by applying singular value decomposition to the un-centered covariance matrix of all input features of previous tasks for each linear layer. For efficiency, the uncentered covariance matrix can be incrementally computed after learning each task. We also empirically verify the rationality of the approximate null space at each linear layer. We apply our approach to training networks for continual learning on benchmark datasets of CIFAR-100 and TinyImageNet, and the results suggest that the proposed approach outperforms or matches the state-ot-the-art continual learning approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “Furthermore, their approaches are dual to each other and are the two sides of the same coin: Wang et al. (2021b) (approximately) maintains a basis for ∩t−1i=1null(X (j) i ) and Saha et al. (2021) for the orthogonal complement of ∩t−1i=1null(X (j) i ); see Appendix G for another example of this…”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “With memory buffers, regularization-based methods can better utilize the old activation knowledge (Rebuffi et al., 2017; Buzzega et al., 2020) and old gradient information (Lopez-Paz & Ranzato, 2017; Wang et al., 2021; Farajtabar et al., 2020).”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Methods such as GEM [34], A-GEM [10], OGD [14] and NSCL [50] fall under this category.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “In addition, there exist other kinds of methods like gradient-based methods (e.g., [56, 73, 16, 58, 61, 64]), regularization-based ones (e.g. [32, 74, 2, 1]), and expansion-based ones (e.g. [69, 34, 25]).”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “Speciﬁcally, we use the uncentered feature covariance, i.e., ˜ X lt = ( X lt ) (cid:62) X lt , to compute the current candidate null space [49].”
Incremental Task Learning with Incremental Rank Updates (2022, methodology): “AdamNSCL [37] gets better results on CIFAR100, but it requires 11.”
Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning (2021, methodology): “Two classic setups of Incremental Learning are Class In-cremental Learning (CIL) [1,12,13,19,20,22,26] and Task Incremental Learning (TIL) [2, 16, 21, 24, 28, 30, 31].”
Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector (2021, neutral): “(Please refer to Algorithm 2 in [37] for more details of obtaining the feature covariance and projection matrix).”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “[44] pointed out that mapping the update of the model parameters for learning the new task into the null space of the previous tasks can help to mitigate catastrophic forgetting.”
Warping the Space: Weight Space Rotation for Class-Incremental Few-Shot Learning (2023, methodology): “Here, note that the activations of deep neural networks are known to have low-rankness property as shown in (Suzuki et al., 2020b; Wang et al., 2021), i.e., singular values decrease rapidly and most of the singular values are close to 0 .”
A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials (2022, methodology): “For instance, [75] proposes a null space CIL (NSCIL) method to train the network on the new task by projecting its gradient updates to the null space of the approximated covariance matrix on all the past task data.”
Continual Variational Autoencoder Learning via Online Cooperative Memorization (2022, methodology): “One of the most popular approaches is to use a regularization loss within the optimization procedure [14,21,23,26,36,41,47,52,56], where the network parameters which are important to the past learnt data are re-weighted when learning a new task, in order to attempt to preserve past knowledge.”
Deep Class-Incremental Learning: A Survey (2023, methodology): “For example, Adam-NSCL [61] proposes to sequentially optimize network parameters by projecting the candidate parameter update into the approximate null space of all previous tasks.”
Learn from Others and Be Yourself in Heterogeneous Federated Learning (2022, background): “Existing continual learning works on tackling catastrophic forgetting can be broadly divided into three branches [10]: replay methods [1, 67], regularization-based methods [32, 47, 53, 85] and parameter isolation methods [55, 61, 70].”
Class-Incremental Learning: A Survey (2023, methodology): “For example, Adam-NSCL [55] proposes to sequentially optimize network parameters by projecting the candidate parameter update into the approximate null space of all previous tasks.”