Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex

By Qianli Liao, Tomaso A. Poggio. 2016

268 citations recorded by the source corpus.

We discuss relations between Residual Networks (ResNet), Recurrent Neural Networks (RNNs) and the primate visual cortex. We begin with the observation that a shallow RNN is exactly equivalent to a very deep ResNet with weight sharing among the layers. A direct implementation of such a RNN, although having orders of magnitude fewer parameters, leads to a performance similar to the corresponding ResNet. We propose 1) a generalization of both RNN and ResNet architectures and 2) the conjecture that a class of moderately deep RNNs is a biologically-plausible model of the ventral stream in visual cortex. We demonstrate the eectiveness of the architectures by testing them on the CIFAR-10 dataset.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Can Active Memory Replace Attention? (2016, background): “In [15] it was shown that the weights of different layers of a residual network can be tied (so it becomes recurrent), without degrading performance.”
Dual Path Networks (2017, background): “Inspired by [12] which demonstrates the relations between the residual networks and RNNs, we prove that the residual networks are densely connected networks when connections are shared across layers.”
Recurrent Batch Normalization (2016, background): “Liao & Poggio (2016) simultaneously investigated batch normalization in recurrent neural networks, albeit only for very short sequences (10 steps).”
Pre-training Graph Neural Networks with Kernels (2018, neutral): “There is an interesting recent work about the parallel between skip-connections (residual networks in that case) and recurrent networks [Liao and Poggio, 2016].”
Active Long Term Memory Networks (2016, background): “The recent interest in bridging human and machine computations [5, 6, 7, 8] obliges to consider a learning framework that is continual, sequential in nature and potentially lifelong [9, 10].”
Spatially Adaptive Computation Time for Residual Networks (2016, methodology): “We show that ACT can be applied to dynamically choose the number of evaluated layers in Residual Network [15, 16] (the similarity between Residual Networks and RNNs was explored in [29, 13]).”
Column Networks for Collective Classification (2016, methodology): “For deep neural nets, following the latest results in (Liao and Poggio 2016; Pham et al. 2016), we implemented high-way network with shared parameters among layers (HWN-noRel).”
Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering (2017, positive): “Since the type of recurrence induced by soft ordering does not require task input or output to be sequential, methods that use recurrence in such a setting are of particular interest (Liang and Hu, 2015; Liao and Poggio, 2016; Pinheiro and Collobert, 2014; Socher et al., 2011; Zamir et al., 2016).”
Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes (2016, methodology): “Several approaches have been proposed to make Batch Normalization applicable to RNNs. Cooijmans et al. (2016) and Liao & Poggio (2016) collect separate batch statistics for each time step.”
Toward an Integration of Deep Learning and Neuroscience (2016, background): “Some have gone on to suggest a direct correspondence between cortical circuits and optimized, appropriately regularized (Sussillo et al., 2015), recurrent neural networks (Liao and Poggio, 2016).”
Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs (2019, background): “Liao & Poggio [23] argued that ResNets can be thought of as recurrent neural networks unrolled over time with non-shared weights, and demonstrated the first working version of a folded ResNet, also explored by [13].”
Learning Functions: When Is Deep Better Than Shallow (2016, background): “Visual cortex has a similar compositional architecture with receptive ﬁelds becoming larger and larger in higher and higher visual areas, with each area corresponding to a recurrent layer in a deep neural network ([Liao and Poggio, 2016]).”
Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks (2021, background): “Iterative processes underlie the human ability to solve sequential reasoning problems, such as complex question answering, proof writing, and even object classiﬁcation [Liao and Poggio, 2016, Kar et al., 2019].”
On Filter Size in Graph Convolutional Networks (2018, background): “1: Graph convolution as described in [8], and adopted with some variations by many state-of-the-art Graph Convolutional neural networks. and recurrent networks [13].”
Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior (2018, result): “This is also consistent with a previous study (Liao and Poggio, 2016) where it was shown that a shallow recurrent neural network (RNN) is equivalent to a very deep CNN (e.g. ResNet) with weight sharing among the layers.”
Convolutional Networks with Dense Connectivity (2019, background): “This makes the state of ResNets similar to (unrolled) recurrent neural networks [10], but the number of parameters of ResNets is G.”