Residual Networks of Residual Networks: Multilevel Residual Networks

By Ke Zhang, Miao Sun, Tony X. Han, Xingfang Yuan, Liru Guo, Tao Liu. 2016

372 citations recorded by the source corpus.

A residual networks family with hundreds or even thousands of layers dominates major image recognition tasks, but building a network by simply stacking residual blocks inevitably limits its optimization ability. This paper proposes a novel residual network architecture, residual networks of residual networks (RoR), to dig the optimization ability of residual networks. RoR substitutes optimizing residual mapping of residual mapping for optimizing original residual mapping. In particular, RoR adds levelwise shortcut connections upon original residual networks to promote the learning capability of residual networks. More importantly, RoR can be applied to various kinds of residual networks (ResNets, Pre-ResNets, and WRN) and significantly boost their performance. Our experiments demonstrate the effectiveness and versatility of RoR, where it achieves the best performance in all residual-network-like structures. Our RoR-3-WRN58-4 + SD models achieve new state-of-the-art results on CIFAR-10, CIFAR-100, and SVHN, with the test errors of 3.77%, 19.73%, and 1.59%, respectively. RoR-3 models also achieve state-of-the-art results compared with ResNets on the ImageNet data set.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

HyperNetworks (2016, methodology): “The original resnet and subsequent derivatives (Zhang et al., 2016; Huang et al., 2016a) achieved state-of-the-art image recognition performance on a variety of public datasets.”
SGDR: Stochastic Gradient Descent with Warm Restarts (2016, background): “Alternative network structures should be also considered; e.g., soon after our initial arXiv report (Loshchilov & Hutter, 2016), Zhang et al. (2016); Huang et al. (2016b); Han et al. (2016) reported that WRNs models can be replaced by more memory-efﬁcient models.”
PolyNet: A Pursuit of Structural Diversity in Very Deep Networks (2016, methodology): “This construction is similar to the building blocks used in the Multiple Residual Network [1].”
Recent advances in convolutional neural networks (2015, methodology): “ residual networks, where each block of RiR contains residual units and non-residual blocks. The RiR can learn how many convolutional layers it should use per residual block. ResNets of ResNets (RoR) [128] is a modication to the ResNets architecture which proposes to use multi-level shortcut connect”
On the Origin of Deep Learning (2017, positive): “For example, Zhang et al. (2016b) took a step further and put Residual Block inside Residual Block.”
The Shallow End: Empowering Shallower Deep-Convolutional Networks through Auxiliary Outputs (2016, background): “Zhang et al. [33] propose a new residual-network architecture called ResNet of ResNet, which add shortcut connections between multilevel blocks in ResNet and achieve better performance.”
Learning for Disparity Estimation Through Feature Constancy (2017, neutral): “It was demonstrated that this architecture outperformed several networks, such as the base network from MC-CNN [32], the conventional high-way network [30], ResNets [7], DenseNet [10], and the ResNets of ResNets [34].”
Short-Term Load Forecasting With Deep Residual Networks (2018, background): “In addition to the stacked residual blocks, extra shortcut connections can be added into the deep residual network, as is introduced in [30].”
SGDR: Stochastic Gradient Descent with Restarts (2016, result): “Alternative network structures should be also considered; e.g., Zhang et al. (2016) just reported results that are basically the same as our best ones (with T 0 = 10, T mult = 2) on CIFAR-10 and (only) one percent worse than ours on CIFAR-100.”
A Robust Deep-Learning-Based Detector for Real-Time Tomato Plant Diseases and Pests Recognition (2017, background): “Since the success of AlexNet [19] in the ImageNet Large Scale Visual Recognition Challenge [20] 2012 (ILSVRC), deeper and deeper networks [21–26] have been proposed and achieved state-of-the-art performance on ImageNet and other benchmark datasets [27].”