Knowledge Transfer in Deep Block-Modular Neural Networks

By Alexander V. Terekhov, Guglielmo Montone, J. O'Regan. 2015

78 citations recorded by the source corpus.

Although deep neural networks DNNs have demonstrated impressive results during the last decade, they remain highly specialized tools, which are trained --- often from scratch --- to solve each particular task. The human brain, in contrast, significantly re-uses existing capacities when learning to solve new tasks. In the current study we explore a block-modular architecture for DNNs, which allows parts of the existing network to be re-used to solve a new task without a decrease in performance when solving the original task. We show that networks with such architectures can outperform networks trained from scratch, or perform comparably, while having to learn nearly 10 times fewer weights than the networks trained from scratch.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

Growing a Brain: Fine-Tuning by Increasing Model Capacity (2017, background): “Interestingly, [34, 38] focus on widening a target network to be twice as large as the source one, but ﬁne-tune only the new units.”
Progressive Neural Networks (2016, methodology): “Details of the experimental setup are provided in section 3 of the Appendix.”
Learning without Forgetting (2016, background): “[21] proposes Deep Block-Modular Neural Networks for fullyconnected neural networks.”
Embracing Change: Continual Learning in Deep Neural Networks. (2020, background): “Recently, a number of adaptive network architectures have emerged that add new capacity while protecting and reusing existing representations [34,35]; this, however, introduces new challenges, such as increasing computational requirements throughout learning.”
Residual Continual Learning (2020, background): “Without this condition, a network can be expanded while keeping the entire original network, e.g., (Terekhov, Montone, and ORegan 2015; Rusu et al. 2016).”
End-to-End Incremental Learning (2018, neutral): “Alternative strategies to mitigate catastrophic forgetting include, increasing the number of layers in the network to learn features for the new classes [28, 32], or slowing”
Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence (2018, background): “One way to address catastrophic forgetting is by dynamically expanding the network for each new task [24,18,20,23].”
Expert Gate: Lifelong Learning with a Network of Experts (2016, background): “Like us, two recent architectures, namely the progressive network [31] and the modular block network [34], also use multiple networks, a new one for each new task.”
Lifelong Generative Modeling (2017, methodology): “of leveraging our model for tasks such as unsupervised learning or even semi-supervised learning Xu et al. (2017). Variational Autoencoders allow us to generate data by sampling the prior and running the sample through the decoder.”
Efficient Parametrization of Multi-domain Deep Neural Networks (2018, background): “Recent work [29, 26] address (a) series (b) parallel”
Superposition of many models into one (2019, background): “Others freeze and grow parts of the network (Terekhov et al., 2015; Rusu et al., 2016; Zenke et al., 2017) to actively preserve previous models during training.”
A review of modularization techniques in artificial neural networks (2019, methodology): “In another work (Terekhov et al., 2015), a similar approach utilized synaptic integration for multi-task learning, where an initial network was trained on some task, followed by a modular block of neurons that were added and integrated by training on a diﬀerent task, while again freezing the…”
Cognitively-Inspired Model for Incremental Learning Using a Few Examples (2020, background): “An alternative set of approaches increase the number of layers in the network for learning new classes [44, 46].”
Continual Learning in Neural Networks (2019, background): “In this family, two architectures, namely the progressive network [134] and the modular block network [150], also use multiple networks, a new one for each new task.”
Incremental Learning Using a Grow-and-Prune Paradigm With Efficient Neural Networks (2019, background): “also propose an algorithm to reuse existing network capacity and add new blocks of neurons, where the generated model outperforms the baseline network that is trained from scratch [23].”
Variational Federated Multi-Task Learning (2019, methodology): “…method can be tuned for a particular application by modifying e.g. the architecture of the lateral connections between devices (Block-Modular NN [42], NinN architecture [27]), the topology of the Bayesian network (star shape, hierarchical etc.), the choice of the variational inference algorithm.”
Small Sample Learning in Big Data Era (2018, methodology): “The block-modular architecture (Terekhov et al., 2015) is similar work while more focused on a visual discrimination task.”
iLGaCo: Incremental Learning of Gait Covariate Factors (2020, methodology): “[35] for incremental learning of object detectors. Other approaches have been presented to mitigate catastrophic forgetting, like increasing the number of layers in the network to learn new knowledge [33, 39], or reducing the learning rate selectively through per-parameter regularization [20]. Rebuf”