Layerwise Optimization by Gradient Decomposition for Continual Learning

By Shixiang Tang, Dapeng Chen, Jinguo Zhu, Shijie Yu, Wanli Ouyang. 2021

94 citations recorded by the source corpus.

Deep neural networks achieve state-of-the-art and sometimes super-human performance across various domains. However, when learning tasks sequentially, the networks easily forget the knowledge of previous tasks, known as "catastrophic forgetting". To achieve the consistencies between the old tasks and the new task, one effective solution is to modify the gradient for update. Previous methods enforce independent gradient constraints for different tasks, while we consider these gradients contain complex information, and propose to leverage inter-task information by gradient decomposition. In particular, the gradient of an old task is decomposed into a part shared by all old tasks and a part specific to that task. The gradient for update should be close to the gradient of the new task, consistent with the gradients shared by all old tasks, and orthogonal to the space spanned by the gradients specific to the old tasks. In this way, our approach encourages common knowledge consolidation without impairing the task-specific knowledge. Furthermore, the optimization is performed for the gradients of each layer separately rather than the concatenation of all gradients as in previous works. This effectively avoids the influence of the magnitude variation of the gradients in different layers. Extensive experiments validate the effectiveness of both gradient-decomposed optimization and layer-wise updates. Our proposed method achieves state-of-the-art results on various benchmarks of continual learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “LOGD [401] decomposes the gradient of each task into task-sharing and task-specific components to leverage inter-task information.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “In addition, there exist other kinds of methods like gradient-based methods (e.g., [56, 73, 16, 58, 61, 64]), regularization-based ones (e.g. [32, 74, 2, 1]), and expansion-based ones (e.g. [69, 34, 25]).”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “These method can be roughly divided into four categories: architecture-based methods expand the network or allocate new neurons for new tasks [13,28,42,60]; replayedbased methods interleave old data with current data by storing historical data in a buffer or generating virtual old data [4,8,12,39,40”
Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “Online continual learning baselines use ResNet18 [35] as backbone and cross-entropy as the classification loss, following [14,68,6,9,13,71].”
Incremental Task Learning with Incremental Rank Updates (2022, methodology): “Memory-based approaches [27,28,8,9,35] usually use memory and replay/rehearsal mechanism to recall a small episodic memory of previous tasks while training new tasks thus reduce the loss in the previous tasks.”
Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning (2021, methodology): “Two classic setups of Incremental Learning are Class In-cremental Learning (CIL) [1,12,13,19,20,22,26] and Task Incremental Learning (TIL) [2, 16, 21, 24, 28, 30, 31].”
Continually Learning Self-Supervised Representations with Projected Functional Regularization (2021, background): “This can be done by either regularizing the weight space (constraining important parameters) [57, 60] or the functional space (constraining predictions or intermediate features) [13, 19, 31].”
Continual Learning with Lifelong Vision Transformer (2022, methodology): “Continual learning baselines use ResNet18 [33] as backbone, and cross-entropy as classification loss, following [7, 10, 14, 15, 67, 71].”
A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials (2022, methodology): “1 Gradient-based methods (e.g., [62, 85, 23, 65, 70, 75]) overcome catastrophic forgetting by minimizing the interference among task-wise gradients when updating network weights.”
Class-Incremental Learning: A Survey (2023, background): “LOGD [57] further decomposes the gradients into shared and task-specific ones.”
Continual Variational Autoencoder Learning via Online Cooperative Memorization (2022, methodology): “One of the most popular approaches is to use a regularization loss within the optimization procedure [14,21,23,26,36,41,47,52,56], where the network parameters which are important to the past learnt data are re-weighted when learning a new task, in order to attempt to preserve past knowledge.”
Deep Class-Incremental Learning: A Survey (2023, background): “LOGD [63] further decomposes the gradients into shared and taskspecific ones.”
Complementary Relation Contrastive Distillation (2021, background): “As gradients measure the fastest rate and direction for loss minimization, gradient relation can explore the structural information of optimization kinetics in representational space [18, 39].”