Neurogenesis deep learning: Extending deep networks to accommodate new classes

By T. Draelos, N. Miner, Christopher C. Lamb, Jonathan A. Cox, C. Vineyard, Kristofor D. Carlson, William M. Severa, C. James, James Bradley Aimone. 2016

105 citations recorded by the source corpus.

Neural machine learning methods, such as deep neural networks (DNN), have achieved remarkable success in a number of complex data processing tasks. These methods have arguably had their strongest impact on tasks such as image and audio processing — data processing domains in which humans have long held clear advantages over conventional algorithms. In contrast to biological neural systems, which are capable of learning continuously, deep artificial networks have a limited ability for incorporating new information in an already trained network. As a result, methods for continuous learning are potentially highly impactful in enabling the application of deep networks to dynamic data sets. Here, inspired by the process of adult neurogenesis in the hippocampus, we explore the potential for adding new neurons to deep layers of artificial neural networks in order to facilitate their acquisition of novel information while preserving previously trained data representations. Our results on the MNIST handwritten digit dataset and the NIST SD 19 dataset, which includes lower and upper case letters and digits, demonstrate that neurogenesis is well suited for addressing the stability-plasticity dilemma that has long challenged adaptive machine learning algorithms.

Subfields: Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

Citations in the corpus

FearNet: Brain-Inspired Model for Incremental Learning (2017, methodology): “This idea was revived in Draelos et al. (2017), where a generative autoencoder was used to create pseudo-examples for unsupervised incremental learning.”
Closed-Loop GAN for continual Learning (2018, background): “Region growing approaches, while usually providing a clean solution for constrained incremental problems, can quickly become memory expensive since they requires both an architectural expansion as well as the storage of at least a portion of old data for retraining [Draelos et al., 2017].”
Learngene: From Open-World to Your Learning Task (2021, methodology): “There are three mainstream methods in deep learning including regularization approaches [38, 29, 32, 72, 31], dynamic-architectures approaches [50, 69, 14, 49], and memory-replay approaches [57, 54, 40, 31, 30].”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “Furthermore, Gepperth and Karaoguz (2016), Kemker and Kanan (2018), and Draelos et al. (2017) integrate neurogenesis into their models where new neurons are created to form new memories.”
Orthogonal Gradient Descent for Continual Learning (2019, methodology): “2017) used auto-encoder to dynamically decide to add neurons for samples with high loss and whether the older data needs to be re-trained or not.”
Continual Unsupervised Representation Learning (2019, background): “Expansion-based models dynamically increase capacity to allow for additional tasks (Rusu et al., 2016; Yoon et al., 2017; Draelos et al., 2017), and optionally prune the network to constrain capacity (Zhou et al., 2012; Golkar et al., 2019).”
Continual Learning for Robotics (2019, methodology): “These methods have also been associated with the term pseudo-rehearsal (Robins, 1995) or Intrinsic Replay (Draelos et al., 2017).”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, neutral): “Neurogenesis deep learning to accommodate new classes (NDL) (Draelos et al., 2017) and lifelong learning with Dynamically Expandable Networks (DEN) (Yoon et al., 2018) have adapted this heuristic approach for use in continual deep learning.”
Closed-Loop Memory GAN for Continual Learning (2018, background): “ for constrained incremental problems, can quickly become memory expensive since they requires both an architectural expansion as well as the storage of at least a portion of old data for retraining [Draelos et al., 2017]. Likewise, replay algorithms run into scalability issues as well. Currently, t”
DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, neutral): “DEMIX-DAPT is related to model expansion techniques in reinforcement learning or vision (Rusu et al., 2016; Draelos et al., 2017) and adapters for pretrained LMs (Houlsby et al.”
Learn-Prune-Share for Lifelong Learning (2020, methodology): “), to accommodate new tasks, while preserving parts of the model for old tasks [14]–[16].”
Continual Learning for Affective Robotics: Why, What and How? (2020, background): “resources can be allocated to extend the capacity, either by expanding trainable parameters [41] or allowing the architecture itself to grow [42] to account for the increased”
FFNB: Forgetting-Free Neural Blocks for Deep Continual Learning (2021, methodology): “,T10 correspond to classes [51-100] while T0 is the pretraining task involving classes [1-50]).”
Biological underpinnings for lifelong learning machines (2022, background): “While neurogenesis allows systems to incorporate new information 201 , uncontrolled growth needs to be avoided.”