Lifelong GAN: Continual Learning for Conditional Image Generation

By Mengyao Zhai, Lei Chen, Frederick Tung, Jiawei He, Megha Nawhal, Greg Mori. 2019

210 citations recorded by the source corpus.

Lifelong learning is challenging for deep neural networks due to their susceptibility to catastrophic forgetting. Catastrophic forgetting occurs when a trained network is not able to maintain its ability to accomplish previously learned tasks when it is trained to perform new tasks. We study the problem of lifelong learning for generative models, extending a trained network to new conditional generation tasks without forgetting previous tasks, while assuming access to the training data for the current task only. In contrast to state-of-the-art memory replay based approaches which are limited to label-conditioned image generation tasks, a more generic framework for continual learning of generative models under different conditional image generation settings is proposed in this paper. Lifelong GAN employs knowledge distillation to transfer learned knowledge from previous networks to the new network. This makes it possible to perform image-conditioned generation tasks in a lifelong learning setting. We validate Lifelong GAN for both image-conditioned and label-conditioned generation tasks, and provide qualitative and quantitative results to show the generality and effectiveness of our method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “For conditional generation, the generated data of previously-learned conditions and their output values are regularized to be consistent between the teacher and student models, such as MeRGANs [125], DRI [129] and LifelongGAN [126].”
Similarity-Preserving Knowledge Distillation (2019, positive): “One way to prevent catastrophic forgetting is to supervise the new model (the student) with the model trained for previous tasks (the teacher) via a knowledge distillation loss [42].”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Since the data of the previous tasks are inaccessible, these methods typically enforce distillation objectives on the current data [32, 12], data from an exemplar memory [19, 13, 42, 7, 2, 27, 23], external data [30] or synthetic data [59].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “Ostapenko et al. [212] combined the HAT method [176] with the generative replay based method using a modified auxiliary classifier GAN [214] architecture.”
Online Coreset Selection for Rehearsal-based Continual Learning (2021, neutral): “In light of this need, continual learning (CL) [40], or lifelong learning, tackles a learning scenario where a model continuously learns over a sequence of tasks [18, 25] within a broad research area, such as classification [15, 9], image generation [47], language learning [24, 4], clinical applicat”
Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation (2021, background): “For paired image generation, we also compare our approach with Lifelong GAN [42] and Progressive Network (PN) [28].”
Training Networks in Null Space of Feature Covariance for Continual Learning (2021, neutral): “Due to inaccessibility to the full datasets of previous tasks, they respectively use data of current task [10, 26], external data [23], coreset of previous tasks [29, 43] or synthetic data [55], resulting in distributional shift [27] from the original datasets.”
Few-Shot Lifelong Learning (2021, methodology): “Some methods utilize generative models to generate data for the old tasks instead of storing the exemplars (Shin et al. 2017; Wu et al. 2018; Zhai et al. 2019; Xiang et al. 2019).”
GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning (2021, methodology): “…the latter retrain part of it; (iii) Rehearsal based aims at preventing catastrophic forgetting by storing and replaying information from previous episodes (Rebuffi et al., 2017; Castro et al., 2018; Wu et al., 2018; Hou et al., 2019; Zhai et al., 2019; Liu et al., 2020b; jun Liu et al., 2020).”
Learning latent representations across multiple data domains using Lifelong VAEGAN (2020, methodology): “e Evaluation. In the following we evaluate numerically the image representation ability for the proposed approach. We consider the reconstruction (Rec) as MSE, and classication accuracy (Acc), as in [67]. The classication accuracy is calculated by a classier which was trained on the generated images”
GAN Memory with No Forgetting (2020, background): “Lifelong GAN [96] employs cycle consistency (via an auxiliary encoder) and knowledge distillation (via copies of”
Knowledge Distillation: A Survey (2020, background): “ge distillation can be well uniﬁed into many kinds of mainstream machine learning, such as the adversarial learning (Liu et al., 2018b), auto machine learning (Macko et al., 2019), lifelong learning (Zhai et al., 2019), reinforcement learning (Ashok et al., 2017), and so on. Thus, the extensions of ”
Few-Shot Class-Incremental Learning (2020, background): “Another way is to use the generative models to memorize the old tasks data distribution [37, 46, 50].”
Lifelong Language Knowledge Distillation (2020, background): “Knowledge distillation has been introduced to the ﬁeld of lifelong learning; for example, Learning without Forgetting (LwF) (Li and Hoiem, 2017), Generative Replay with Distillation (DGR+distill), Replay-through-Feedback (RtF) (van de Ven and Tolias, 2018), and Lifelong GAN (Zhai et al., 2019), a lo”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, methodology): “Both continual generative modeling, Shin et al. (2017); Achille et al. (2018); Farquhar and Gal (2018a); Nguyen et al. (2018); Wu et al. (2018); Zhai et al. (2019), as well as the bulk of class incremental continuous learning works Li and Hoiem (2016); Kirkpatrick et al. (2017); Rebuﬃ et al.…”
Explaining How Deep Neural Networks Forget by Deep Visualization (2020, unknown): “Catastrophic Forgetting Many works have managed to address the forgetting problem in generative models [19], object detection [13], semantic segmentation [15], or captioning [12].”
LFS-GAN: Lifelong Few-Shot Image Generation (2023, background): “One such approach is Lifelong GAN [50], which uses a knowledge distillation [16] based technique to prevent catastrophic forgetting.”