Greedy Layerwise Learning Can Scale to ImageNet

By Eugene Belilovsky, Michael Eickenberg, Edouard Oyallon. 2018

219 citations recorded by the source corpus.

Shallow supervised 1-hidden layer neural networks have a number of favorable properties that make them easier to interpret, analyze, and optimize than their deep counterparts, but lack their representational power. Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks. Contrary to previous approaches using shallow networks, we focus on problems where deep learning is reported as critical for success. We thus study CNNs on image classification tasks using the large-scale ImageNet dataset and the CIFAR-10 dataset. Using a simple set of ideas for architecture and training we find that solving sequential 1-hidden-layer auxiliary problems lead to a CNN that exceeds AlexNet performance on ImageNet. Extending this training methodology to construct individual layers by solving 2-and-3-hidden layer auxiliary problems, we obtain an 11-layer network that exceeds several members of the VGG model family on ImageNet, and can train a VGG-11 model to the same accuracy as end-to-end learning. To our knowledge, this is the first competitive alternative to end-to-end training of CNNs that can scale to ImageNet. We illustrate several interesting properties of these models theoretically and conduct a range of experiments to study the properties this training induces on the intermediate layers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Why Should We Add Early Exits to Neural Networks? (2020, methodology): “A preliminary formal analysis of the LT approach is provided in [ 8 ] , and later extended in [ 7 ] .”
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking (2018, methodology): “, 2018) and (Belilovsky et al., 2018), authors propose methods to train a DNN sequentially, block-by-block, for re-”
Backward Feature Correction: How Deep Learning Performs Deep Learning (2020, methodology): “ting others to zero, and so on. Such algorithm is used in theoretical works such as [60]. There exist other works that use (deep) auxiliary networks to train the layers of a neural network one by one [13]; the authors of [13] also refer to their algorithm as layer-wise training; but in our language,”
Training Neural Networks with Local Error Signals (2019, methodology): “The best reported result on CIFAR-10 is 7.2% using local classiﬁers and ensembling, approaching the results of global backprop (Belilovsky et al., 2018).”
PPFL: privacy-preserving federated learning with trusted execution environments (2021, methodology): “Furthermore, recently this method was shown to scale for large datasets such as ImageNet and to achieve performance comparable to regular end-to-end ML [5].”
Decoupled Greedy Learning of CNNs (2019, methodology): “The greedy layerwise CNN training procedure of (Belilovsky et al., 2018), which we parallelize, similarly relies on an auxiliary network.”