Residual Networks are Exponential Ensembles of Relatively Shallow Networks

By Andreas Veit, Michael J. Wilber, Serge J. Belongie. 2016

109 citations recorded by the source corpus.

In this work, we introduce a novel interpretation of residual networks showing they are exponential ensembles. This observation is supported by a large-scale lesion study that demonstrates they behave just like ensembles at test time. Subsequently, we perform an analysis showing these ensembles mostly consist of networks that are each relatively shallow. For example, contrary to our expectations, most of the gradient in a residual network with 110 layers comes from an ensemble of very short networks, i.e., only 10-34 layers deep. This suggests that in addition to describing neural networks in terms of width and depth, there is a third dimension: multiplicity, the size of the implicit ensemble. Ultimately, residual networks do not resolve the vanishing gradient problem by preserving gradient flow throughout the entire depth of the network - rather, they avoid the problem simply by ensembling many short networks together. This insight reveals that depth is still an open research question and invites the exploration of the related notion of multiplicity.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Scaling & Learning Theory

Citations in the corpus

Gaussian Error Linear Units (GELUs) (2016, background): “ng. The stochasticity of dropout and zoneout may allow for a pseudo-ensemble [1] of networks with stochastic width or stochastic depth, respectively, and this leads to marked performance improvements [5, 14]. Since ReLUs lack stochasticity and since the aforementioned regularizers are irrespective o”
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation (2016, background): “Recent evidence suggest that ResNets behave like ensemble of relatively shallow networks [35]: ”Residual networks avoid the vanishing gradient problem by introducing short paths which can carry gradient throughout the extent of very deep networks”.”
Bilinear Attention Networks (2018, background): “An interesting property of residual learning is robustness toward arbitrary ablations [31].”
Deep Pyramidal Residual Networks (2016, methodology): “[32], ResNets can be described as ensembles of relatively shallow networks, showing that deleting an individual building block in a residual unit of ResNets affects minor classification loss, whereas removing layers from plain networks such as VGG [25] severely reduces the classification rate.”
PolyNet: A Pursuit of Structural Diversity in Very Deep Networks (2016, background): “Second, as revealed by recent work [29, 12], ResNets, which have obtained top performances in a number of tasks, can be viewed as an implicit ensemble of shallow networks.”
Robust Large Margin Deep Neural Networks (2016, positive): “This observation is consistent with the claims in [49], which states that ResNets resemble an ensemble of relatively shallow networks.”
Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations (2017, positive): “We also note that, our viewpoint enables us to easily explain why ResNet can achieve good accuracy by dropping out some residual blocks after training, whereas dropping off sub-sampling layers often leads to an accuracy drop (Veit et al., 2016).”
Toward an Integration of Deep Learning and Neuroscience (2016, background): “…network architectures that are superﬁcially very deep, but which possess particular types of “skip connections,” can actually be seen as ensembles of comparatively shallow networks (Veit et al., 2016); applied in the time domain, this could limit the need to propagate errors far backwards in time.”
The Shallow End: Empowering Shallower Deep-Convolutional Networks through Auxiliary Outputs (2016, background): “resents the original supervision information obtained from all the auxiliary losses. However, when Lis very large, since ResNet can be considered as an ensemble of exponentially many shallow networks [29], the term ( P K j=k j P L j 1 i=l @ x l F(x i;W i)) in g 3 can be exponentially large and domin”
On the Compactness, Efficiency, and Representation of 3D Convolutional Networks: Brain Parcellation as a Pretext Task (2017, background): “The unravelled view of the residual connections proposed by Veit et al. [20] suggests that the networks with n residual blocks have a collection of 2 n unique paths.”
Hard-Aware Deeply Cascaded Embedding (2016, background): “[36] that ResNet is actually exponential ensembles of relatively shallow networks.”
The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification (2017, background): “[Veit et al., 2016] discusses ResNet, a state-of-the-art network structure, could be understood as an exponential ensembles of shallow networks.”
MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network (2017, background): “Recent work [32] shows that ResNet with n residual blocks can be interpreted as the ensemble of 2 relatively shallow networks.”
Residual Connections Encourage Iterative Inference (2017, background): “…has given rise to two school of thoughts about Resnets– the ensemble view where they are thought to learn an exponential ensemble of shallower models Veit et al. (2016), and the unrolled iterative estimation view Liao & Poggio (2016); Greff et al. (2016), where Resnet layers are thought to…”
A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning (2017, background): “Interestingly, the deep residual network has shortcut connections to make an ensemble structure [24].”
Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation (2018, unknown): “Similar experimental results were reported in [29].”