Residual Connections Encourage Iterative Inference

By Stanislaw Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, Yoshua Bengio. 2017

182 citations recorded by the source corpus.

Residual networks (Resnets) have become a prominent architecture in deep learning. However, a comprehensive understanding of Resnets is still a topic of ongoing research. A recent view argues that Resnets perform iterative refinement of features. We attempt to further expose properties of this aspect. To this end, we study Resnets both analytically and empirically. We formalize the notion of iterative refinement in Resnets by showing that residual connections naturally encourage features of residual blocks to move along the negative gradient of loss as we go from one block to the next. In addition, our empirical analysis suggests that Resnets are able to perform both representation learning and iterative refinement. In general, a Resnet block tends to concentrate representation learning behavior in the first few layers while higher layers perform iterative refinement of features. Finally we observe that sharing residual layers naively leads to representation explosion and counterintuitively, overfitting, and we show that simple existing strategies can help alleviating this problem.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization (2018, background): “As indicated in (Jastrzebski et al. (2018)), most intense changes of the main branch ﬂow of ResNet are concentrated after reduction block.”
Architecture Matters in Continual Learning (2022, background): “…optima; Bartlett et al. (2018b) study the function expressivity of residual architectures; Bartlett et al. (2018a) show that gradient descent provably learns linear transforms in ResNets; Jastrzebski et al. (2017) show that skip connections tend to iteratively reﬁne the learned representations.”
Understanding intermediate layers using linear classifier probes (2016, methodology): “This is connected to the work done by Jastrzebski et al. (2017).”
Eliciting Latent Predictions from Transformers with the Tuned Lens (2023, methodology): “Jastrz˛ebski et al. (2017) argue that skip connections encourage neural networks to perform “iterative inference” in the following sense: each layer reliably updates the hidden state in a direction of decreasing loss.”
NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations (2018, background): “A recent interpretation is that these feed-forward architectures implement iterative inference [13, 23].”
Multi-level Residual Networks from Dynamical Systems View (2017, background): “These views provide preliminary interpretations, however, deep understanding of ResNets is still an active on-going research topic (Jastrzebski et al., 2017; Li et al., 2016; Hardt & Ma, 2017; Li & Yuan, 2017).”
Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs (2019, background): “Liao & Poggio [23] argued that ResNets can be thought of as recurrent neural networks unrolled over time with non-shared weights, and demonstrated the first working version of a folded ResNet, also explored by [13].”
An investigation of model-free planning (2019, background): “Although RNNs (and more recently ResNets) can in principle learn a function that can be iterated to obtain a result (Graves, 2016; Jastrzebski et al., 2017; Greff et al., 2016), it is not clear whether the networks trained in our RL domains learn to amortize computation over time in this way.”
Functional Gradient Boosting based on Residual Network Perception (2018, methodology): “Note that this formulation explains the optimization view (Jastrzebski et al., 2017) of ResNet mathematically.”
Dynamic Recursive Neural Network (2019, methodology): “Recently, based on the studies about recurrent struc ture and iterative refinement [24, 28], densely connected structure is widely used to obtain more efficient model like DenseNet [18] and CliqueNet [43].”
Transformers learn to implement preconditioned gradient descent for in-context learning (2023, background): “The seminal work by Jastrzebski et al. (2018) observes that hidden representations across the layers of deep neural networks approximately implement gradient descent.”
Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023, background): “Figure 6 visualizes OthelloGPT’s “iterative infer-ence” (Jastrzebski et al., 2018; Belrose et al., 2023; Veit et al., 2016; nostalgebraist, 2020), or iterative reﬁnement of features.”
IamNN: Iterative and Adaptive Mobile Neural Network for Efficient Image Classification (2018, methodology): “The same approach was used in Jastrzebski et al. (2017) in their experiments with shared weights.”
A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay (2018, background): “Smith (2015) tested numerous ways to vary the learning rate between the two boundary values, found them to be equivalent and therefore recommended the simplest, which is letting the learning rate change linearly (Jastrzebski et al. (2017b) suggest discrete jumps and obtained similar results).”
Convolutional Neural Networks with Alternately Updated Clique (2018, background): “Recent studies show that the skip connection mechanism can be extrapolated as a recurrent neural network (RNN) or LSTM [14], when weights are shared among different layers [27, 5, 21].”