Net2Net: Accelerating Learning via Knowledge Transfer

By Tianqi Chen, Ian J. Goodfellow, Jonathon Shlens. 2015

751 citations recorded by the source corpus.

We introduce techniques for rapidly transferring the information stored in one neural net into another neural net. The main purpose is to accelerate the training of a significantly larger neural net. During real-world workflows, one often trains very many different neural networks during the experimentation and design process. This is a wasteful process in which each new model is trained from scratch. Our Net2Net technique accelerates the experimentation process by instantaneously transferring the knowledge from a previous network to each new deeper or wider network. Our techniques are based on the concept of function-preserving transformations between neural network specifications. This differs from previous approaches to pre-training that altered the function represented by a neural net when adding layers to it. Using our knowledge transfer mechanism to add depth to Inception modules, we demonstrate a new state of the art accuracy rating on the ImageNet dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding (2019, background): “…a process of distilling or transferring the knowledge from a (set of) large, cumbersome model(s) to a lighter, easier-to-deploy single model, without signiﬁcant loss in performance (Bucilu et al., 2006; Hinton et al., 2015; Balan et al., 2015; Ba et al., 2016; Chen et al., 2015; Tan et al., 2019).”
AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, methodology): “We will show that GauInit obtains the best result, challenging the idea of Network Morphism [5, 36, 37].”
Wide Residual Networks (2016, background): “We call the resulting network structures wide residual networks (WRNs) and show that these are far superior over their commonly used thin and very deep counterparts.”
Learning without Forgetting (2016, methodology): “The weights from all nodes at previous layer to the new nodes at current layer are initialized the same way Net2Net [5] would expand a layer by copying nodes.”
Patient Knowledge Distillation for BERT Model Compression (2019, background): “Chen et al. (2015) introduced techniques for efﬁciently transferring knowledge from an existing network to a deeper or wider network.”
AutoML: A Survey of the State-of-the-Art (2019, background): “The network morphism based algorithms [13], [18], [119] can also inherit the weights of previous architectures.”
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, methodology): “, 2018), we make use of operators acting on the space of neural network architectures that preserve the function a network represents, dubbed network morphisms (Chen et al., 2015; Wei et al., 2016), obviating training from scratch and thereby substantially reducing the required training time per net”
Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples (2016, background): “Learning substitutes is an instance of knowledge transfer, a set of techniques to transfer the generalization knowledge learned by a model into another model [9, 10].”
A Comprehensive Survey of Neural Architecture Search (2020, methodology): “Net2Net [46] conducted a detailed study of network transformation technology and proposed the function-preserving transformations to Fig.”
Fast Neural Network Adaptation via Parameter Remapping and Architecture Search (2020, background): “Parameter Remapping Net2Net (Chen et al., 2015) proposes the function-preserving transformations to remap the parameters of one network to a new deeper or wider network.”
Network Pruning via Transformable Architecture Search (2019, background): “in different searching candidates is the same. TAS is a differentiable NAS method, which is able to efﬁciently search for a transformable networks with ﬂexible width and depth. Network transformation [5, 10, 3] also studied the depth and width of networks. Chen et al. [5] manually widen and deepen a”
EENA: Efficient Evolution of Neural Architecture (2019, methodology): “However, faced with the scale of contemporary neural architectures with millions of weights, recent approaches based on evolutionary algorithm have been imporved in some ways, such as using gradient-based methods for optimizing weights [29, 28, 23] or modifying the architecture by network morphisms ”
A Survey on Neural Architecture Search (2019, background): “Both actions are performed by means of functionpreserving transformations (Chen et al., 2016).”
Overcoming Multi-Model Forgetting (2019, neutral): “EAS (Cai et al., 2018) also uses an RL strategy to grow the network depth or layer width with function-preserving transformations deﬁned by Chen et al. (2016) where they initialize new models with previous parameters.”
Auto-Keras: An Efficient Neural Architecture Search System (2018, methodology): “It is a technique to morph the architecture of a neural network but keep its functionality [10, 45].”
Path-Level Network Transformation for Efficient Architecture Search (2018, methodology): “For example, Cai et al. (2018) utilized Net2Net (Chen et al., 2016) operations, a class of function-preserving transformation operations, to further ﬁnd high-performance architectures based on a given network, while Ashok et al. (2018) used network compression operations to compress well-trained…”
AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, background): “[6] proposed to speed up the exploration via network transformation [8].”
SMASH: One-Shot Model Architecture Search through HyperNetworks (2017, methodology): “Another simple technique (which our code nominally supports) is using the HyperNet-generated weights to initialize the resulting network and accelerate training, similar to Net2Net (Chen et al., 2016).”