Generative Models from the perspective of Continual Learning

By Timothée Lesort, Hugo Caselles-Dupré, Michaël Garcia Ortiz, Andrei Stoian, David Filliat. 2018

171 citations recorded by the source corpus.

Which generative model is the most suitable for Continual Learning? This paper aims at evaluating and comparing generative models on disjoint sequential image generation tasks. We investigate how several models learn and forget, considering various strategies: rehearsal, regularization, generative replay and fine-tuning. We used two quantitative metrics to estimate the generation quality and memory ability. We experiment with sequential tasks on three commonly used benchmarks for Continual Learning (MNIST, Fashion MNIST and CIFAR10). We found that among all models, the original GAN performs best and among Continual Learning strategies, generative replay outperforms all other methods. Even if we found satisfactory combinations on MNIST and Fashion MNIST, training generative models sequentially on CIFAR10 is particularly instable, and remains a challenge. Our code is available online1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Meta & Continual Learning

Citations in the corpus

DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “Another method we also exploit is to ﬁrst learn a state representation [Lesort et al., 2018] to compress the observation into a low dimensional embedding and secondly learn the policy on top of this representation.”
CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, methodology): “Regularizing knowledge in the prediction space is done using knowledge distillation [22] and it has been found to perform better than model regularization-based methods for continual learning when task labels are not given [37, 61].”
FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, background): “Many works [10,15,20,33,37] create new modules to handle the growing training distribution [26,41] dynamically.”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “, pixels) [59, 5, 10] but can also be compressed [29, 35], or trimmed [18] to reduce memory overhead; others store only a model to generate new samples of past classes [37, 66, 45].”
Continual Learning via Local Module Composition (2021, methodology): “Continual learning methods typically address the problem of forgetting through parameter regularization [46, 70, 99], replay [89, 78, 3, 73, 54, 12, 97, 39, 81, 13] or dynamic architectures (and MoEs) [83, 87, 52, 85, 53, 41].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “Lesort et al. [133] investigated the performance of different generative models including Variational AutoEncoder (VAE), conditional VAE, conditional GAN, Wasserstein GAN, and Wasserstein GAN with Gradient Penalty.”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “Prediction space regularization (accomplished using knowledge distillation) has been found to perform better than model regularization based methods for class-incremental learning [33, 55].”
Optimizing Reusable Knowledge for Continual Learning via Metalearning (2021, background): “Other works such as [33, 34] train generator functions (GANs) or autoencoders [35] to”
Continual Learning by Using Information of Each Class Holistically (2021, methodology): “…learn to generate pseudo old data and use them to jointly train with the new task data (Shin et al. 2017; Wu et al. 2018; Kamra, Gupta, and Liu 2017; Seff et al. 2017; Hu et al. 2019; Lesort et al. 2018; Ostapenko et al. 2019; Hayes et al. 2019) (also known as rehearsal (von Oswald et al. 2020)).”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “Although replay has been more extensively explored in supervised classification and reinforcement learning, it has also been explored in unsupervised learning settings (Lesort et al., 2019; Wu et al., 2018).”
Online Continual Learning in Image Classification: An Empirical Survey (2021, background): “Generative replay [115, 47] is another direction but it is not viable for more complex datasets as the current deep generative models still cannot generate satisfactory images for such datasets [17, 50].”
Curriculum-Meta Learning for Order-Robust Continual Relation Extraction (2021, methodology): “(ii) Dynamic architecture methods (Lesort et al. 2019; Mallya, Davis, and Lazebnik 2018) dynamically expand model architectures to learn new tasks and effectively prevent forgetting of old tasks.”
Do not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting (2021, methodology): “Other data-generation based methods [23, 33, 38, 39, 42] use generative adversarial network [9] and by replying the total data, new model is trained.”
Meta-Consolidation for Continual Learning (2020, background): “One line of work [88, 38, 47, 2, 16, 71, 56, 13, 79, 42] constrains the parameters of the deep network trained on Task A to not change much while learning a new Task B, while another - replay-based methods - store [63, 48, 14, 66, 9, 3, 4, 22] or generate [74, 81, 46, 45] examples of previous tasks ”
Online Class-Incremental Continual Learning with Adversarial Shapley Value (2020, methodology): “…methods that utilize a generative model to produce virtual samples instead of using a memory buffer (Shin et al. 2017), recent research has demonstrated the limitations of such approaches with convolutional neural networks in datasets such as CIFAR-10 (Aljundi et al. 2019a; Lesort et al. 2019).”
GAN Memory with No Forgetting (2020, methodology): “Among these methods, generative replay is believed an effective and general strategy for lifelong learning problems [66, 88, 40, 78], as discussed above.”
BooVAE: Boosting Approach for Continual Learning of VAE (2019, neutral): “(Lesort et al., 2019) compare EwC, generative replay and random coresets.”
Online Continual Learning with Maximally Interfered Retrieval (2019, background): “[23] found that generative replay is not yet a viable strategy for CIFAR-10 given the current state of the generative modeling.”