Network Morphism

By Tao Wei, Changhu Wang, Yong Rui, Chang Wen Chen. 2016

195 citations recorded by the source corpus.

We present a systematic study on how to morph a well-trained neural network to a new one so that its network function can be completely preserved. We define this as network morphism in this research. After morphing a parent network, the child network is expected to inherit the knowledge from its parent network and also has the potential to continue growing into a more powerful one with much shortened training time. The first requirement for this network morphism is its ability to handle diverse morphing types of networks, including changes of depth, width, kernel size, and even subnet. To meet this requirement, we first introduce the network morphism equations, and then develop novel morphing algorithms for all these morphing types for both classic and convolutional neural networks. The second requirement is its ability to deal with non-linearity in a network. We propose a family of parametric-activation functions to facilitate the morphing of any continuous nonlinear activation neurons. Experimental results on benchmark datasets and typical neural networks demonstrate the effectiveness of the proposed network morphism scheme.

Subfields: Statistical and Nonlinear Physics, Model Architectures, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical and Nonlinear Physics

In Model Architectures

In Neural Architecture Search & AutoML

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “sets and layer architectures. There are some previous works which add or morph layers to increase the depth in DNNs. VggNet [32] and DropIn [33] added new layers into shallower DNNs; Network Morphism [5, 36, 37] morphed each layer to multiple layers to increase the depth meanwhile preserving the fun”
AutoML: A Survey of the State-of-the-Art (2019, background): “The network morphism based algorithms [13], [14] can also inherit the weights of previous architectures.”
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “For example, Wei et al. (2016) use a special case of this operator to deal with non-linear, non-idempotent activation functions.”
EENA: Efficient Evolution of Neural Architecture (2019, methodology): “However, faced with the scale of contemporary neural architectures with millions of weights, recent approaches based on evolutionary algorithm have been imporved in some ways, such as using gradient-based methods for optimizing weights [29, 28, 23] or modifying the architecture by network morphisms ”
Random Search and Reproducibility for Neural Architecture Search (2019, background): “New approaches introduce a significant amount of algorithmic complexity in the search process, including complicated training routines [1, 39, 45, 7], architecture transformations [44, 41, 6, 30, 11], and modeling assumptions [21, 24, 48, 5, 29] (see Figure 1 and Appendix A.”
Scalable Recollections for Continual Lifelong Learning (2017, background): “As a result, our work is related in motivation to techniques that look to preserve knowledge after transforming the network architecture (Chen et al., 2015; Wei et al., 2016).”
Simple And Efficient Architecture Search for Convolutional Neural Networks (2017, methodology): “Wei et al. (2016) proposed additional operations, e.g., for handling non-idempotent activation functions or altering the kernel size and introduced the term network morphism.”
NSGA-NET: A Multi-Objective Genetic Algorithm for Neural Architecture Search (2018, methodology): “Elsken et al. Thomas Elsken (2018) use a simple hill climbing method along with a network morphism Wei et al. (2016b) approach to optimize network architectures quickly on limited resources.”
SP-NAS: Serial-to-Parallel Backbone Search for Object Detection (2020, methodology): “As shown in Figure 1, for the “expand” operation, we add a new block between two existing blocks by IdMorph [43] while the number of input/output channels remains the same.”
Online Deep Learning: Learning Deep Neural Networks on the Fly (2017, methodology): “Some do this explicitly by Growing of Networks via the function preservation principle [Chen et al., 2016; Wei et al., 2016], where the (student) network of higher capacity is at least as good as the shallower (teacher) network.”
A Survey on Evolutionary Neural Architecture Search (2020, background): “[181] introduced two function-preserving operators for DNNs, and these operators are termed as network morphisms [182].”
Incremental Training of Deep Convolutional Neural Networks (2018, background): “Other steps towards incremental training are presented in [10,11], where the goal is to transfer knowledge from a small network towards a significantly larger network under some architectural constraints.”
Language Models with Transformers (2019, background): “Recent studies (Wei et al., 2016; Cai et al., 2018; Elsken et al., 2017) focus on extending the set of the network transformations to handle additional operations such as non-linear activation functions and skip connections.”
An Equivalence of Fully Connected Layer and Convolutional Layer (2017, methodology): “…to FC layers to analyze the properties of CONV layers in the equivalent FC layers, e.g. uncertainty in CONV layers (Gal, 2016), (Gal et al., 2017), (Blundell et al., 2015), or we can apply the methods in FC layers into CONV layers, e.g. network morphism (Chen et al., 2015), (Wei et al., 2016).”
Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks (2021, methodology): “Network Morphism (Wei et al., 2016) extends the Net2Net to a broader concept, which deﬁnes more operations that change a network’s architecture but maintains its functional representation.”
SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers (2019, methodology): “Like Elsken et al. [23], SpArSe uses a form of weight sharing called network morphism [62] to search over architectures without training each one from scratch.”
Embedding Watermarks into Deep Neural Networks (2017, background): “In [4, 32], a systematic study has been done on how to morph a well-trained neural network into a new one so that its network function can be completely preserved for further training.”