Scalable and Order-robust Continual Learning with Additive Parameter Decomposition

By Jaehong Yoon, Saehoon Kim, Eunho Yang, Sung Ju Hwang. 2019

211 citations recorded by the source corpus.

While recent continual learning methods largely alleviate the catastrophic problem on toy-sized datasets, some issues remain to be tackled to apply them to real-world problem domains. First, a continual learning model should effectively handle catastrophic forgetting and be efficient to train even with a large number of tasks. Secondly, it needs to tackle the problem of order-sensitivity, where the performance of the tasks largely varies based on the order of the task arrival sequence, as it may cause serious problems where fairness plays a critical role (e.g. medical diagnosis). To tackle these practical challenges, we propose a novel continual learning method that is scalable as well as order-robust, which instead of learning a completely shared set of weights, represents the parameters for each task as a sum of task-shared and sparse task-adaptive parameters. With our Additive Parameter Decomposition (APD), the task-adaptive parameters for earlier tasks remain mostly unaffected, where we update them only to reflect the changes made to the task-shared parameters. This decomposition of parameters effectively prevents catastrophic forgetting and order-sensitivity, while being computation- and memory-efficient. Further, we can achieve even better scalability with APD using hierarchical knowledge consolidation, which clusters the task-adaptive parameters to obtain hierarchically shared parameters. We validate our network with APD, APD-Net, on multiple benchmark datasets against state-of-the-art continual learning methods, which it largely outperforms in accuracy, scalability, and order-robustness.

Subfields: Artificial Intelligence, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Besides, the network parameters themselves can be decomposed into task-sharing and task-specific elements, such as by additive decomposition (APD [479]), singular value decomposition (RCM [197]), filter atom decomposition (FAS [300]) and lowrank factorization (IBP-WF [294], IRU [177]).”
A Theoretical Study on Solving Continual Learning (2022, background): “ADP [56] decomposes parameters into shared and adaptive parts to construct an order robust TIL system.”
Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer (2022, methodology): “, [24, 16, 31, 30, 26, 29]) allocate different model parameters to different tasks, fix the parameters for old tasks by masking them out during the new task learning and expand the network when needed.”
Class Gradient Projection For Continual Learning (2022, methodology): “The 10-Split CIFAR-100 is constructed by randomly splitting the 100 classes of CIFAR-100 into 10 tasks with 10 classes per task.”
DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, methodology): “Parameter isolation methods [29,44] update the architecture in order to isolate the knowledge learned from each task in order to prevent forgetting.”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “Architecture-based methods allocate diﬀerent parameters or add new parameters for the new task, while sharing parameters across tasks to reduce the interference of previous tasks [3, 22, 27–29, 41, 42, 44, 53, 56, 60].”
TRGP: Trust Region Gradient Projection for Continual Learning (2022, methodology): “We further compare with Progressive Neural Network (PNN) (Rusu et al., 2016), Learning Without Forgetting (LWF) (Li & Hoiem, 2017), Dynamic-Expansion Net (DEN) (Yoon et al., 2017), and APD (Yoon et al., 2020), by using CIFAR-100 Sup dataset.”
Continual Learning with Recursive Gradient Optimization (2022, methodology): “…utilizes a tiny ring memory to alleviate forgetting, (6) GPM (Saha et al., 2021) which trains new tasks in the residual gradient subspace, (7) APD (Yoon et al., 2019) which is a strong expansion-based method decomposing the parameters of different tasks with a common basis, and (8) STL which…”
AFEC: Active Forgetting of Negative Transfer in Continual Learning (2021, neutral): “(1) CIFAR-100-SC [35]: CIFAR-100 can be split as 20 superclasses (SC) with 5 classes per superclass dependent on semantic similarity, where each superclass is a classiﬁcation task.”
Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference (2021, methodology): “(3) Parameter isolation methods augment the networks (Rusu et al., 2016; Lee et al., 2017a; Aljundi et al., 2017) or decompose the network into subnetworks for each task (Mallya & Lazebnik, 2018; Cheung et al., 2019; Yoon et al., 2020).”
Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning (2021, methodology): “† and ∗ denotes results reported by [13] and [44] respectively.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[148] proposed a method called Additive Parameter Decomposition (APD).”
Model Zoo: A Growing Brain That Learns Continually (2021, background): “…(Zenke et al., 2017), Permuted-MNIST (Kirkpatrick et al., 2017), Split-CIFAR10 (Zenke et al., 2017), Split-CIFAR100 ∗ (Zenke et al., 2017), Coarse-CIFAR100 (Rosenbaum et al., 2017; Yoon et al., 2019; Shanahan et al., 2021) and Split-miniImagenet (Vinyals et al., 2016; Chaudhry et al., 2019b).”
Online Coreset Selection for Rehearsal-based Continual Learning (2021, positive): “The architecture approaches [33, 44, 43, 23, 45] utilize network isolation or expansion during continual learning to improve network performance.”
Rectification-based Knowledge Retention for Continual Learning (2021, methodology): “Our method utilizes task labels during testing similar to [26, 35, 36].”
Gradient Projection Memory for Continual Learning (2021, methodology): “For this experiment, similar to APD (Yoon et al., 2020), we use a modified LeNet-5 architecture with 20-50-800-500 neurons.”
Curriculum-Meta Learning for Order-Robust Continual Relation Extraction (2021, background): “fairness in the medical domain (Yoon et al. 2020); (ii) bench-marking of continual learning algorithms as most of the existing works pick an arbitrary and random sequence of the given tasks for evaluation (Chen and Liu 2016); (iii) uncertainty to the quality of extracted knowledge in the realistic s”
Federated Continual Learning with Weighted Inter-client Transfer (2020, methodology): “This decomposition scheme is motivated by (9).”