Adversarial Continual Learning

By Sayna Ebrahimi, F. Meier, Roberto Calandra, Trevor Darrell, Marcus Rohrbach. 2020

236 citations recorded by the source corpus.

Continual learning aims to learn new tasks without forgetting previously learned ones. We hypothesize that representations learned to solve each task in a sequence have a shared structure while containing some task-specific properties. We show that shared features are significantly less prone to forgetting and propose a novel hybrid continual learning framework that learns a disjoint representation for task-invariant and task-specific features required to solve a sequence of tasks. Our model combines architecture growth to prevent forgetting of task-specific skills and an experience replay approach to preserve shared skills. We demonstrate our hybrid approach is effective in avoiding forgetting and show it is superior to both architecture-based and memory-based approaches on class incrementally learning of a single dataset as well as a sequence of multiple datasets in image classification. Our code is available at \url{this https URL}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Infectious Diseases, Safety, Robustness & Interpretability, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Infectious Diseases

In Safety, Robustness & Interpretability

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “For a regular network, the task-speciﬁc components could be parallel branches [233], [234], adaptive layers [58], [235], masks of intermediate features [186], [236], [237].”
CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, background): “One group of approaches expands a model’s architecture as new tasks are encountered; these are highly effective for applications where a model growing with tasks is practical [16, 35, 39, 43, 54].”
Class Gradient Projection For Continual Learning (2022, methodology): “[35], we evaluate on several continual learning benchmarks, including 10-Split CIFAR-100 [20], 20-Split CIFAR-100 [20], 5-Split CIFAR-100 [20], CIFAR-100 Superclass , and sequence of 5-Datasets [13].”
A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learning (2022, methodology): “85 Table 2: Continual learning performance expressed in Average Accuracy and Forgetting at the end of the learning sequence of the 5-dataset benchmark [6].”
Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, background): “Many works mitigate catastrophic forgetting during the learning process [37,14,48,68,31,42,20,2,10,3,64].”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, methodology): “However, they often introduce a substantial number of additional parameters and sometimes involve simplified assumption like known test time task identity [11, 35, 34], which falls into the setting of task-incremental learning.”
Representation Compensation Networks for Continual Semantic Segmentation (2022, neutral): “In the more challenging scenario without any replay, many methods explore regularization to maintain old knowledge, including knowledge distillation [10, 18, 21, 28, 51, 69, 74, 100], adversarial training [29,88], vanilla regularization [43,48,56,66,67,80,95,98] and so on.”
Class-Incremental Continual Learning Into the eXtended DER-Verse (2022, neutral): “each comprises of 10 classes with 500 and 100 32 32 images each for training and testing respectively; Split miniImageNet [15], [47], [48] derives from miniImageNet [49], a 100-class subset of the popular ImageNet dataset, split in 20 classification tasks.”
An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, neutral): “(Ebrahimi et al., 2020) consists of ﬁve 10-way image classiﬁcation datasets: CIFAR-10 (Krizhevsky & Hin-ton, 2009), MNIST (LeCun, 1998), Fashion-MNIST (Xiao et al., 2017), SVHN (Netzer et al., 2011), and notMNIST (Bulatov, 2011).”
Learning to Prompt for Continual Learning (2021, methodology): “We use Split CIFAR-100 [22] and 5datasets [12] for class-incremental setting, CORe50 [30] for domain-incremental setting, and Gaussian scheduled CIFAR-100 [52] for task-agnostic setting, to evaluate the effectiveness of our method.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “The middle nodes of the tree represent the parameters that are shared by a subset of the tasks.”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “For example, methods which expand a model’s architecture as new tasks are encountered are highly effective for applications where a model growing with tasks is practical [12, 32, 36, 40, 48, 51].”
Optimizing Reusable Knowledge for Continual Learning via Metalearning (2021, methodology): “Numbers for HAT, A-GEM and ACL are taken from the respective articles.”
Gradient Projection Memory for Continual Learning (2021, methodology): “Datasets: We evaluate our continual learning algorithm on Permuted MNIST (PMNIST) (Le-cun et al., 1998), 10-Split CIFAR-100 (Krizhevsky, 2009), 20-Spilt miniImageNet (Vinyals et al., 2016) and sequence of 5-Datasets (Ebrahimi et al., 2020b).”
Do not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting (2021, methodology): “The adversarial continual learning [7] method learns a disjoint representation for task-invariant and task-specific features.”
Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting (2020, methodology): “This simple method, known as experience replay, has been explored and shown to be effective (Rebufﬁ et al., 2017; Wu et al., 2019; Castro et al., 2018; Rajasegaran et al., 2020; Ebrahimi et al., 2020b; Hayes et al., 2019; Riemer et al., 2018).”
MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, neutral): “Recently, there has been a large body of research in incremental learning [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43].”
Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, background): “ACL [70] fuses a dynamic architecture with exemplars, explicitly disentangling shared and task-specific features with an adversarial loss.”
Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer (2021, neutral): “Architectural approaches typically avoid over-writing the current model by expanding the model parameters to make room for knowledge related to novel tasks [44, 45, 46, 47, 48, 49].”
Continual Learning via Bit-Level Information Preserving (2021, methodology): “Among these methods, HAT and ACL incur linearly growing memory overheads due to task specific parameters.”