Residual Networks Behave Like Ensembles of Relatively Shallow Networks

By Andreas Veit, Michael J. Wilber, Serge J. Belongie. 2016

1,142 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Scaling Laws for Neural Language Models (2020, background): “In [VWB16] it was argued that deep models can function as ensembles of shallower models, which could potentially explain this ﬁnding.”
Aggregated Residual Transformations for Deep Neural Networks (2016, background): “[40] interpret a single ResNet as an ensemble of shallower networks, which results from ResNet’s additive behaviors [15].”
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth (2021, methodology): “Skip connections were first introduced in ResNets [HZRS16a], ever since, it has been used to facilitate optimization in deep networks [HZRS16b, VWB16b, BFL + 18].”
Deep Pyramidal Residual Networks (2016, methodology): “units. this phenomenon is related to the overall improvement in the classiﬁcation performance enabled by stochastic depth. Motivated by the ensemble interpretation of residual networks in Veit et al. [33] and the results with stochastic depth [10], we devised another method to handle the phenomenon ”
Path-Level Network Transformation for Efficient Architecture Search (2018, background): “According to the research of Veit et al. (2016), ResNets can be considered to behave as ensembles of a collection of many paths of differing length.”
Understanding intermediate layers using linear classifier probes (2016, background): “papers seeking to understand better how they work (Veit et al., 2016; Larsson et al., 2016; Singh et al., 2016).”
Wider or Deeper: Revisiting the ResNet Model for Visual Recognition (2016, methodology): “6 in [31].”
The Shattered Gradients Problem: If resnets are the answer, then what is the question? (2017, background): “The closest work to ours is Veit et al. (2016), which shows resnets behave like ensembles of shallow networks.”
Transformers without Tears: Improving the Normalization of Self-Attention (2019, background): “We speculate this is related to identity residual networks acting like shallow ensembles (Veit et al., 2016) and thus undermining the learning of the longest path; further study is required.”
Data-Driven Sparse Structure Selection for Deep Neural Networks (2017, background): “[39] interpreted ResNet as an exponential ensemble of many shallow networks.”
How to Start Training: The Effect of Initialization and Architecture (2018, methodology): “To state our results about FM1 for ResNets, we must set some notation (based on the framework presented e.g. in Veit et al. [30]).”
Convolutional Networks with Adaptive Inference Graphs (2017, background): “Lukas Boschert 10 [2]ResNets behave like ensembles ResNet: Residual Network”
Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering (2017, background): “The single-task case results in a model with increased commutativity between layers, a behavior that has also been observed in residual networks (Veit et al., 2016), whereas here the result is a set of layers that are assembled in different ways for different tasks .”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, background): “[50] show that ResNets are resilient to layer dropping at test time, which motivates our approach; however, they do not provide a way to dynamically choose which layers could be removed from a network without sacrificing accuracy.”