Modularized Morphing of Neural Networks

By Tao Wei, Changhu Wang, Chang Wen Chen. 2017

21 citations recorded by the source corpus.

In this work we study the problem of network morphism, an effective learning scheme to morph a well-trained neural network to a new one with the network function completely preserved. Different from existing work where basic morphing types on the layer level were addressed, we target at the central problem of network morphism at a higher level, i.e., how a convolutional layer can be morphed into an arbitrary module of a neural network. To simplify the representation of a network, we abstract a module as a graph with blobs as vertices and convolutional layers as edges, based on which the morphing process is able to be formulated as a graph transformation problem. Two atomic morphing operations are introduced to compose the graphs, based on which modules are classified into two families, i.e., simple morphable modules and complex modules. We present practical morphing solutions for both of these two families, and prove that any reasonable module can be morphed from a single convolutional layer. Extensive experiments have been conducted based on the state-of-the-art ResNet on benchmark datasets, and the effectiveness of the proposed solution has been verified.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Jiawei Han
2. Chen Chen
3. Jing Li
4. Hongkun Yu
5. Liyuan Liu
6. Deng Cai
7. Charu C. Aggarwal
8. Xiaofei He

In Electrical and Electronic Engineering

1. Jiawei Han

In Natural Language Processing

1. Jiawei Han
2. Hongkun Yu
3. Liyuan Liu

Citations in the corpus

• AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, critical): “We will show that GauInit obtains the best result, challenging the idea of Network Morphism [5, 36, 37].”
• On the Transformer Growth for Progressive BERT Training (2020, background): “Network Morphism (Wei et al., 2016, 2017) manages to grow a layer to multiple layers with the represented function intact.”
• Forward Thinking: Building and Training Neural Networks One Layer at a Time (2017, background): “In particular several recent papers have elements that can be nicely described as special cases of forward thinking; see for example, net2net [3], cascade correlation [4], network morphism [17, 16], and convolutional representation transfer [11].”