A Strategy for an Uncompromising Incremental Learner

By Ragav Venkatesan, Hemanth Venkateswara, Sethuraman Panchanathan, Baoxin Li. 2017

42 citations recorded by the source corpus.

Multi-class supervised learning systems require the knowledge of the entire range of labels they predict. Often when learnt incrementally, they suffer from catastrophic forgetting. To avoid this, generous leeways have to be made to the philosophy of incremental learning that either forces a part of the machine to not learn, or to retrain the machine again with a selection of the historic data. While these hacks work to various degrees, they do not adhere to the spirit of incremental learning. In this article, we redefine incremental learning with stringent conditions that do not allow for any undesirable relaxations and assumptions. We design a strategy involving generative models and the distillation of dark knowledge as a means of hallucinating data along with appropriate targets from past distributions. We call this technique, phantom sampling.We show that phantom sampling helps avoid catastrophic forgetting during incremental learning. Using an implementation based on deep neural networks, we demonstrate that phantom sampling dramatically avoids catastrophic forgetting. We apply these strategies to competitive multi-class incremental learning of deep neural networks. Using various benchmark datasets and through our strategy, we demonstrate that strict incremental learning could be achieved. We further put our strategy to test on challenging cases, including cross-domain increments and incrementing on a novel label space. We also propose a trivial extension to unbounded-continual learning and identify potential for future development.

Subfields: Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

1. Bing Liu
2. Jiangpeng He
3. Bernt Schiele
4. Feng Zhu
5. Timothée Lesort
6. Yaoyao Liu
7. Zixuan Ke
8. Gyuhak Kim

Citations in the corpus

• Overcoming catastrophic forgetting with hard attention to the task (2018, neutral): “Within the pseudo-rehearsal category, we could also consider recent approaches that substitute the memory module by a generative network (Venkatesan et al., 2017; Shin et al., 2017; Nguyen et al., 2017).”
• Learning with Recoverable Forgetting (2022, background): “Apart from those works that still need part of the old data, many researchers are devoted to developing the methods without storing the old data by synthesizing old data [35, 39] or even without referring to any old data [37, 36, 27].”
• Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks (2021, background): “Other related works include Phantom Sampling (Venkatesan et al., 2017), Conceptor-Aided Backprop (He and Jaeger, 2018), Gating Networks (Masse et al., 2018; Serrà et al., 2018), PackNet (Mallya and Lazebnik, 2018), Diffusion-based Neuromodulation (Velez and Clune, 2017), IMM (Lee et al., 2017),…”
• Incremental Learning in Online Scenario (2020, methodology): “In [26] and [28], synthetic data is used to retain the knowledge for old classes by applying a deep generative model [6].”
• Mnemonics Training: Multi-Class Incremental Learning Without Forgetting (2020, methodology): “Related methods [27, 11, 32] used Generative Adversarial Networks (GAN) [6] to generate old samples in each new phase for data replaying, and good results were obtained in the multi-task incremental setting.”
• Large Scale Incremental Learning (2019, methodology): “[22, 27] highly depends on the capability of generative models and struggles with complex objects and scenes.”
• Three scenarios for continual learning (2019, methodology): “We note that it is possible to combine DGR with distillation by replaying input samples from a generative model and pairing them with soft targets [see also 6, 22].”
• Generative replay with feedback connections as a general strategy for continual learning (2018, methodology): “We note that it is possible to combine LwF and DGR by replaying input samples from a generative model and pairing them with soft targets [see also 6, 18].”
• Generative Models from the perspective of Continual Learning (2018, methodology): “[5, 21], where a generative model is used to produce samples from previous tasks.”
• A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020, background): “me as random selection, with higher computational time. 7.3. Incremental learning without memory In many applications, no memory of past classes is available. For instance, in medical data processing [72], this is often due to privacy issues. We study the behavior of the main algorithms which can be”
• Overcoming Catastrophic Forgetting for Continual Learning via Model Adaptation (2018, background): “, 2017), phantom sampling (Venkatesan et al., 2017), active long term memory networks (Furlanello et al.”
• Unsupervised Continual Learning Via Pseudo Labels (2021, methodology): “Based on data type, there are pseudo rehearsal based methods [44, 48, 20, 52], which use synthetic or embeddings of data, and methods use original data as exemplars [39, 31, 8, 28, 36, 37, 2, 4, 10, 9].”