Wide Residual Networks
By Sergey Zagoruyko, Nikos Komodakis. 2016
8,993 citations recorded by the source corpus.
Deep residual networks were shown to be able to scale up to thousands of layers and still have improving performance. However, each fraction of a percent of improved accuracy costs nearly doubling the number of layers, and so training very deep residual networks has a problem of diminishing feature reuse, which makes these networks very slow to train. To tackle these problems, in this paper we conduct a detailed experimental study on the architecture of ResNet blocks, based on which we propose a novel architecture where we decrease depth and increase width of residual networks. We call the resulting network structures wide residual networks (WRNs) and show that these are far superior over their commonly used thin and very deep counterparts. For example, we demonstrate that even a simple 16-layer-deep wide residual network outperforms in accuracy and efficiency all previous deep residual networks, including thousand-layer-deep networks, achieving new state-of-the-art results on CIFAR, SVHN, COCO, and significant improvements on ImageNet. Our code and models are available at this https URL
Subfields: Artificial Intelligence, Civil and Structural Engineering, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Sergey Levine
- Dacheng Tao
- Ruslan Salakhutdinov
In Civil and Structural Engineering
- Quoc V. Le
- Ekin Dogus Cubuk
- Barret Zoph
- Samuel L. Smith
- Frank Hutter
- Shuicheng Yan
- Nojun Kwak
- Dan Hendrycks
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Yi Yang
- Alan L. Yuille
In Model Architectures
- Quoc V. Le
- Yoshua Bengio
- Alan L. Yuille
- Jiashi Feng
- Shuicheng Yan
- Barret Zoph
- Jianfeng Gao
- Dacheng Tao
Citations in the corpus
- Scaling Laws for Neural Language Models (2020, background): “Earlier work [ZK16] has compared width and depth, and found that wide ResNets can outperform deep ResNets on image classification.”
- Denoising Diffusion Probabilistic Models (2020, methodology): “Our neural network architecture follows the backbone of PixelCNN++ [49], which is a U-Net [45] based on a Wide ResNet [64].”
- A Constructive Prediction of the Generalization Error Across Scales (2019, methodology): “For image classification, we train ResNet (He et al., 2016) and WRN models (Zagoruyko & Komodakis, 2016) with stochastic gradient decent (SGD).”
- Unsupervised Data Augmentation for Consistency Training (2019, methodology): “We follow the settings in [46] and employ WideResNet-28-2 [67, 21] as our baseline model.”
- Gaussian Error Linear Units (GELUs) (2016, methodology): “Next we consider a wide residual network on CIFAR-100 with 40 layers and a widening factor of 4 (Zagoruyko & Komodakis, 2016).”
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, background): “However, deeper networks are also more difficult to train due to the vanishing gradient problem (Zagoruyko & Komodakis, 2016).”
- Wide neural networks of any depth evolve as linear models under gradient descent (2019, methodology): “This decrease goes approximately as 1 /N for fully-connected networks, with more ambiguous scaling for convolutional and WRN architectures.”
- The State of Sparsity in Deep Neural Networks (2019, methodology): “To verify our l 0 regularization implementation, we applied our weight-level code to Wide ResNet (Zagoruyko & Komodakis, 2016) trained on CIFAR-10 and replicated the training FLOPs reduction and accuracy re-sults from the original publication.”
- Don't Decay the Learning Rate, Increase the Batch Size (2017, methodology): ““Original training schedule” follows the implementation of Zagoruyko & Komodakis (2016), using an initial learning rate of 0.1 which decays by a factor of 5 at each step, a momentum coefficient of 0.9, and a batch size of 128.”
- Attention Augmented Convolutional Networks (2019, methodology): “Image classification on the CIFAR-100 dataset [22] using the Wide-ResNet 28-10 architecture [51].”
- Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, neutral): “[109] Zagoruyko, S.”
- Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “We also use deeper (50, 101, 152 and 200 layers) and wider (from 1× to 4×) ResNets, as in [54, 44, 8].”
- VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “We train three VirTex models with L = 1, H = 1024 textual heads, and different visual backbones: (a) ResNet-50 (default), (b) ResNet-50 w2× [104] (2× channel width), and (c) ResNet101 (2× depth).”
- Do ImageNet Classifiers Generalize to ImageNet? (2019, neutral): “[61] Sergey Zagoruyko and Nikos Komodakis.”
- HyperNetworks (2016, methodology): “We obtained similar classification accuracy numbers as reported in (Zagoruyko & Komodakis, 2016) with our own implementation.”
- SGDR: Stochastic Gradient Descent with Warm Restarts (2016, methodology): “…state-of-the-art results on CIFAR-10, CIFAR-100, SVHN, Ima-geNet, PASCAL VOC and MS COCO datasets were obtained by Residual Neural Networks (He et al., 2015; Huang et al., 2016c; He et al., 2016; Zagoruyko & Komodakis, 2016) trained without the use of advanced methods such as AdaDelta and Adam.”
- Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “First, we train a family of Wide-ResNet architectures [47], where the model size may be systematically altered through the widening parameter governing the number of convolutional filters.”
- mixup: Beyond Empirical Risk Minimization (2017, methodology): “For WideResNet, we follow (Zagoruyko & Komodakis, 2016a) and divide the learning rate by 10 after 60, 120 and 180 epochs.”
- Feature Pyramid Networks for Object Detection (2016, neutral): “: This entry of AttractioNet [10] adopts VGG-16 for proposals and Wide ResNet [39] for object detection, so is not strictly a single-model result.”