Learning Deep ResNet Blocks Sequentially using Boosting Theory

By Furong Huang, J. Ash, J. Langford, Robert E. Schapire. 2017

120 citations recorded by the source corpus.

Deep neural networks are known to be difficult to train due to the instability of back-propagation. A deep \emph{residual network} (ResNet) with identity loops remedies this by stabilizing gradient computations. We prove a boosting theory for the ResNet architecture. We construct $T$ weak module classifiers, each contains two of the $T$ layers, such that the combined strong learner is a ResNet. Therefore, we introduce an alternative Deep ResNet training algorithm, \emph{BoostResNet}, which is particularly suitable in non-differentiable architectures. Our proposed algorithm merely requires a sequential training of $T$ "shallow ResNets" which are inexpensive. We prove that the training error decays exponentially with the depth $T$ if the \emph{weak module classifiers} that we train perform slightly better than some weak baseline. In other words, we propose a weak learning condition and prove a boosting theory for ResNet under the weak learning condition. Our results apply to general multi-class ResNets. A generalization error bound based on margin theory is proved and suggests ResNet's resistant to overfitting under network with $l_1$ norm bounded weights.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Why Should We Add Early Exits to Neural Networks? (2020, methodology): “, through boosting [17, 24], clustering [44], and kernel similarity [33].”
Progressive Neural Architecture Search (2017, positive): “Finally, [24,25] grow CNNs sequentially using boosting.”
Regularized Evolution for Image Classifier Architecture Search (2018, background): “Other than through RL or evolution, architecture search was also explored with cascade-correlation (Fahlman & Lebiere, 1990), boosting (Cortes et al., 2016; Huang et al., 2017), hillclimbing (Elsken et al.”
Neural Architecture Optimization (2018, background): “Some recent works [17, 18, 26] try to improve the efﬁciency in architecture search by exploring the search space incrementally and sequentially, typically from shallow to hard.”
Auto-Meta: Automated Gradient Based Meta Learner Search (2018, background): “This line of thinking is present in hypernetworks [4], co-evolving NEAT [15], boosting [6, 11], MCTS [17], early stopping of unpromising models [3], and progressive neural architecture search(PNAS) [13].”
Residual Connections Encourage Iterative Inference (2017, background): “Resnets have also been studied from the perspective of boosting theory Huang et al. (2017).”
Ensemble deep learning: A review (2021, background): “The theoretical background for the success of the Deep Residual networks (DeepResNet) [13] was explained in the context of boosting theory [74].”
Decoupled Greedy Learning of CNNs (2019, methodology): “Recently, several authors have revisited the classic (Ivakhnenko & Lapa, 1965; Bengio et al., 2007) approach of supervised greedy layer-wise training of neural networks (Huang et al., 2018; Marquez et al., 2018).”
Greedy Layerwise Learning Can Scale to ImageNet (2018, background): “, 2015) show global optimal approximations, while other works have shown that networks based on 1-hidden layer training can have a variety of guarantees under certain assumptions (Huang et al., 2017; Malach & Shalev-Shwartz, 2018; Arora et al., 2014): greedy layerwise methods could permit to cascade”
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking (2018, methodology): “In (Huang et al., 2018) and (Belilovsky et al., 2018), authors propose methods to train a DNN sequentially, block-by-block, for reducing the training costs.”
Shallow Learning For Deep Networks (2018, background): “…while other works have shown that networks based on 1-hidden layer training can have a variety of guarantees under certain assumptions (Huang et al., 2017; Malach & Shalev-Shwartz, 2018; Arora et al., 2014): greedy layerwise methods could permit to cascade those results to bigger…”
Functional Gradient Boosting based on Residual Network Perception (2018, methodology): “…great success in clas-siﬁcation tasks; in particular, residual network (ResNet) (He et al., 2016) and its variants such as wide-ResNet (Zagoruyko & Komodakis, 2016), ResNeXt (Xie et al., 2017), and DenseNet (Huang et al., 2017b) have become the most prominent architectures in computer vision.”