Gradual DropIn of Layers to Train Very Deep Neural Networks

By Leslie N. Smith, Emily M. Hand, T. Doster. 2015

37 citations recorded by the source corpus.

We introduce the concept of dynamically growing a neural network during training. In particular, an untrainable deep network starts as a trainable shallow network and newly added layers are slowly, organically added during training, thereby increasing the network's depth. This is accomplished by a new layer, which we call DropIn. The DropIn layer starts by passing the output from a previous layer (effectively skipping over the newly added layers), then increasingly including units from the new layers for both feedforward and backpropagation. We show that deep networks, which are untrainable with conventional methods, will converge with DropIn layers interspersed in the architecture. In addition, we demonstrate that DropIn provides regularization during training in an analogous way as dropout. Experiments are described with the MNIST dataset and various expanded LeNet architectures, CIFAR-10 dataset with its architecture expanded from 3 to 11 layers, and on the ImageNet dataset with the AlexNet architecture expanded to 13 layers and the VGG 16-layer architecture.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “re. We will show that AutoGrow generalizes to different datasets and layer architectures. There are some previous works which add or morph layers to increase the depth in DNNs. VggNet [32] and DropIn [33] added new layers into shallower DNNs; Network Morphism [5, 36, 37] morphed each layer to multip”
Deep Networks with Stochastic Depth (2016, methodology): “Many follow up works have been empirically successful, such as DropConnect [20], Maxout [21] and DropIn [22].”
Ensemble deep learning: A review (2021, background): “2016 Gradual dropin of layers to train very deep neural networks [109] Implicit ensemble Classification”
Knowledge Distillation from Few Samples (2018, background): “Similar gradually adjusting trick is also applied to train very-deep neural networks (Smith et al., 2016).”
Regularization for Deep Learning: A Taxonomy (2017, background): “, 2016b), DropIn (Smith et al., 2015) Connections between layers; noise Similar to Dropout: extracting complementary (non-coadapted) features across different levels of abstraction is helpful; implicit model ensemble.”