Deep Residual Learning for Image Recognition
By Kaiming He, X. Zhang, Shaoqing Ren. Computer Vision and Pattern Recognition, 2015
234,194 citations recorded by the source corpus.
Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers - 8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training
View Paper: https://arxiv.org/abs/1512.03385
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Sergey Levine
- Percy Liang
- Luke S. Zettlemoyer
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Model Architectures
- Yoshua Bengio
- Quoc V. Le
- Alan L. Yuille
- Jianfeng Gao
- Ruslan Salakhutdinov
- Kyunghyun Cho
- Dacheng Tao
- Kaiming He
In Optimization & Training
- Peter Richtárik
- Yoshua Bengio
- Quoc V. Le
- Simon Lacoste-Julien
- Sebastian U. Stich
- Martin Jaggi
- Francis R. Bach
- Frank Hutter
Citations in the corpus
- Attention is All you Need (2017, methodology): “We employ a residual connection [10] around each of the two sub-layers, followed by layer normalization [1].”
- Convolutional Sequence to Sequence Learning (2017, methodology): “Our model is equipped with gated linear units (Dauphin et al., 2016) and residual connections (He et al., 2015a).”
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, background): “We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.”
- Neural Machine Translation in Linear Time (2016, methodology): “Each layer is wrapped in a residual block that contains additional convolutional layers with filters of size 1 × 1 (He et al., 2016). residual We adopt two variants of the blocks: one with ReLUs, which is used in the machine translation experiments, and one with Multiplicative Units (Kalchbrenner et ”
- Xception: Deep Learning with Depthwise Separable Convolutions (2016, background): “in [4], which our proposed architecture uses extensively.”
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “Let (X,Y ) be a source and target sentence pair.”
- Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, background): “However, the depth of 16 is not very deep compared to the models in computer vision (He et al., 2016).”
- Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “All systems use residual connections between the contextual layers (He et al., 2016).”
- Universal Language Model Fine-tuning for Text Classification (2018, methodology): “…CV models (including object detection, classification, and segmentation) are rarely trained from scratch, but instead are fine-tuned from models that have been pretrained on ImageNet, MS-COCO, and other datasets (Sharif Razavian et al., 2014; Long et al., 2015a; He et al., 2016; Huang et al., 2017).”
- Learned in Translation: Contextualized Word Vectors (2017, methodology): “[2016] propose to combine text representations with visual representations extracted by a pretrained residual network [He et al., 2016].”
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “For fair comparison, we also report SkipThought results in our setting, using 2048-dimensional pretrained ResNet-101 (He et al., 2016) instead of the original VGG19 features and 113k training images.”
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (2015, background): “to propose regions from data, and thus can easily benefit from deeper and more expressive features (such as the 101layer residual nets adopted in [18]).”
- Object Detection Networks on Convolutional Feature Maps (2015, methodology): “Combining with orthogonal improvements, the results in [17] secured the 1st place in MS COCO and ImageNet 2015 challenges.”
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015, methodology): “We want to a preserve the information in the network, by normalizing the activations in a training example relative to the statistics of the entire training data.”
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “Layer normalization [Ba et al., 2016] is applied to the input of each subcomponent and a residual skip connection [He et al., 2016] adds each subcomponent’s input to its output.”
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, positive): “In computer vision, however, convolutional architectures remain dominant (LeCun et al., 1989; Krizhevsky et al., 2012; He et al., 2016).”
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “In Figure 2 we show that a 63 million parameter transformer language model, which already uses twice the compute of its ResNet-50 image encoder, learns to recognize ImageNet classes three times slower than a much simpler baseline that predicts a bag-ofwords encoding of the same text.”