Decoupled Greedy Learning of CNNs

By Eugene Belilovsky, Michael Eickenberg, Edouard Oyallon. 2019

140 citations recorded by the source corpus.

A commonly cited inefficiency of neural network training by back-propagation is the update locking problem: each layer must wait for the signal to propagate through the full network before updating. Several alternatives that can alleviate this issue have been proposed. In this context, we consider a simpler, but more effective, substitute that uses minimal feedback, which we call Decoupled Greedy Learning (DGL). It is based on a greedy relaxation of the joint training objective, recently shown to be effective in the context of Convolutional Neural Networks (CNNs) on large-scale image classification. We consider an optimization of this objective that permits us to decouple the layer training, allowing for layers or modules in networks to be trained with a potentially linear parallelization in layers. With the use of a replay buffer we show this approach can be extended to asynchronous settings, where modules can operate with possibly large communication delays. We show theoretically and empirically that this approach converges. Then, we empirically find that it can lead to better generalization than sequential greedy optimization. We demonstrate the effectiveness of DGL against alternative approaches on the CIFAR-10 dataset and on the large-scale ImageNet dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Why Should We Add Early Exits to Neural Networks? (2020, background): “In [ 9 ] , the authors investigate this approach for parallel optimization of the network, where a set of intermediate buffers is added to ensure the possibility of decoupling a layer from the previous ones.”
Online Learned Continual Compression with Stacked Quantization Module (2019, methodology): “Notably unlike VQVAE-2 (Razavi et al., 2019), each module is learned greedily without backward communication between modules using the current estimate of z ( i − 1) q similar to Belilovsky et al. (2019); Nøkland & Eidnes (2019).”
Online Learned Continual Compression with Adaptive Quantization Modules (2019, methodology): “We also note, since we maintain stored samples at each module and the modules are decoupled, that such an approach allows to easily distribute training in an asynchronous manner as per Belilovsky et al. (2019).”
Backward Feature Correction: How Deep Learning Performs Deep Learning (2020, result): “uot; of the higher-level signals. This means during training, most of the \backward" eort in our neural network is from the \identity link". This is consistent with empirical works such as [12, 63], while the authors observe that in ResNet, the \backward" from hidden weights can be de”
Pipelined Backpropagation at Scale: Training Large Models without Batches (2020, background): “Unlocking the forward, backward, and update operations is an active area of research (Jaderberg et al., 2017; Huo et al., 2018a;b; Xu et al., 2019; Belilovsky et al., 2019; Gaunt et al., 2017).”