Incremental Classifier Learning with Generative Adversarial Networks

By Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, Zhengyou Zhang, Yun Raymond Fu. 2018

117 citations recorded by the source corpus.

In this paper, we address the incremental classifier learning problem, which suffers from catastrophic forgetting. The main reason for catastrophic forgetting is that the past data are not available during learning. Typical approaches keep some exemplars for the past classes and use distillation regularization to retain the classification capability on the past classes and balance the past and new classes. However, there are four main problems with these approaches. First, the loss function is not efficient for classification. Second, there is unbalance problem between the past and new classes. Third, the size of pre-decided exemplars is usually limited and they might not be distinguishable from unseen new classes. Forth, the exemplars may not be allowed to be kept for a long time due to privacy regulations. To address these problems, we propose (a) a new loss function to combine the cross-entropy loss and distillation loss, (b) a simple way to estimate and remove the unbalance between the old and new classes , and (c) using Generative Adversarial Networks (GANs) to generate historical data and select representative exemplars during generation. We believe that the data generated by GANs have much less privacy issues than real images because GANs do not directly copy any real image patches. We evaluate the proposed method on CIFAR-100, Flower-102, and MS-Celeb-1M-Base datasets and extensive experiments demonstrate the effectiveness of our method.

Subfields: Generative Models, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Meta & Continual Learning

Citations in the corpus

Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “The combination of knowledge distillation and generative replay based methods has also been explored [205–207].”
Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, background): “Knowledge distillation [52, 66, 36, 13] relies on a teacher (old) model transferring or remembering knowledge related to previous tasks to a student model which is trained to learn also additional tasks.”
Online Continual Learning in Image Classification: An Empirical Survey (2021, methodology): “Instead of saving raw samples, an alternative is Generative Replay which trains a deep generative model such as GAN [45] to generate pseudodata that mimic past data for replay [46, 47, 48].”
EEC: Learning to Encode and Regenerate Images for Continual Learning (2021, background): “Note that MeRGAN and EEC approach the JT upperbound on A5.”
Do not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting (2021, methodology): “Other data-generation based methods [23, 33, 38, 39, 42] use generative adversarial network [9] and by replying the total data, new model is trained.”
Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams (2020, methodology): “The replay data is either obtained directly from operational memoryM with stored samples [37, 31, 48, 18] or generated using generative models [42, 23, 39, 50].”
Knowledge Distillation for Incremental Learning in Semantic Segmentation (2019, methodology): “For this reason, some recent methods [2], [31], instead, do not store old classes data but compensate this by training Generative Adversarial Networks (GANs) to generate images containing previous classes when new classes should be learned, thus retaining high accuracy on old tasks.”
Compacting, Picking and Growing for Unforgetting Continual Learning (2019, methodology): “Data-preserving approaches (such as [32, 3, 11, 34]) are designed to directly save important data or latent codes as an efficient form, while memory-replay approaches [41, 46, 13, 45, 27] introduce additional memory models such as GANs for keeping data information or distribution in an indirect way.”
Incremental Learning Techniques for Semantic Segmentation (2019, methodology): “For this reason, some recent methods [29, 35] do not store old data but comar X iv :1 90 7.”
Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning (2019, background): “Second, as pointed out by [32, 22] storing raw samples of previous data can violate data privacy and memory restrictions of real-world applications.”
Overcoming Catastrophic Forgetting With Unlabeled Data in the Wild (2019, background): “We note that other more sophisticated selection algorithms like herding [33] do not perform significantly better than random selection, which is also reported in prior works [3, 42].”
Learning Without Memorizing (2018, methodology): “Therefore, most existing solutions [4, 14, 18] explore incremental learning (IL) by allowing the model to retain a fraction of the training data of base classes, while incrementally learning new classes.”
Generative replay with feedback connections as a general strategy for continual learning (2018, neutral): “However, due to the wide variety of experimental protocols used to evaluate them, many of these methods claim “state-of-the-art” performance (e.g., Kirkpatrick et al., 2017; Rebuffi et al., 2017; Nguyen et al., 2017; Masse et al., 2018; Kemker & Kanan, 2018; Wu et al., 2018).”
Generative Models from the perspective of Continual Learning (2018, background): “As the generator may overfit the few samples kept in memoryand then maximizes the evaluation proposed by [17], while not producing diverse samples.”
Revisiting Distillation and Incremental Classifier Learning (2018, critical): “[25] also tried to independently reproduce the results of herding but failed.”
Continual Learning for Robotics (2019, methodology): “A classical method implementing a generative replay normally makes use of dual models [69, 150, 172, 41, 70].”
Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data (2018, background): “In [25] and [26], instead of using the old data directly, fake previous data are generated by generative adversarial networks.”