Identity Mappings in Deep Residual Networks
By Kaiming He, X. Zhang, Shaoqing Ren. 2016
11,282 citations recorded by the source corpus.
Deep residual networks have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62 % error) and CIFAR-100, and a 200-layer ResNet on ImageNet. Code is available at: https://github.com/KaimingHe/resnet-1k-layers.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Ocean Engineering, Model Architectures, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Sergey Levine
- Jianfeng Gao
- Percy Liang
- Dacheng Tao
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Yi Yang
- Andrew Zisserman
In Ocean Engineering
- Yi Yang
- Tao A. Xiang
- Xiaogang Wang
- Wanli Ouyang
- Ross B. Girshick
- Liang Zheng
- Abhinav Kumar Gupta
- Kaiming He
In Model Architectures
- Quoc V. Le
- Yoshua Bengio
- Alan L. Yuille
- Jiashi Feng
- Ruslan Salakhutdinov
- Kaiming He
- Kyunghyun Cho
- Shuicheng Yan
In Optimization & Training
- Quoc V. Le
- Yoshua Bengio
- Frank Hutter
- Martin Jaggi
- Simon Lacoste-Julien
- Sebastian U. Stich
- Peter Richtárik
- Soham De
Citations in the corpus
- Neural Machine Translation in Linear Time (2016, methodology): “Each layer is wrapped in a residual block that contains additional convolutional layers with filters of size 1 × 1 (He et al., 2016). residual We adopt two variants of the blocks: one with ReLUs, which is used in the machine translation experiments, and one with Multiplicative Units (Kalchbrenner et…”
- Inductive Representation Learning on Large Graphs (2017, positive): “This concatenation can be viewed as a simple form of a “skip connection” [13] between the different “search depths”, or “layers” of the GraphSAGE algorithm, and it leads to significant gains in performance (Section 4).”
- Momentum Contrast for Unsupervised Visual Representation Learning (2019, neutral): “The models in [14] are R101 v2 [34], and others are R50.”
- Generating Long Sequences with Sparse Transformers (2019, methodology): “First, we use the pre-activation residual block of (He et al., 2016), defining a network of N layers in the following way: where embed is a function we describe in the next section, W out is a weight matrix, and resblock( h ) normalizes the input to the attention block and a positionwise feedforward…”
- Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “For Open Images, we chose ResNet-50.”
- Mixed Precision Training (2017, methodology): “He et al. (2016c) propose quantized variants of the GRU (Cho et al., 2014) and Long Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) cells to use fewer bits for weights and activations.”
- Deep Pyramid Convolutional Neural Networks for Text Categorization (2017, methodology): “We empirically observed that pre-activation indeed outperformed ‘post-activation’, which is in line with the image results (He et al., 2016).”
- Wide Residual Networks (2016, methodology): “dual blocks results in an increase of the number of parameters, we studied the effect of dropout to regularize training and prevent overfitting. Previously, dropout in residual networks was studied in [13] with dropout being inserted in the identity part of the block, and the authors showed negative ”
- Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “However, despite many successes in computer vision and natural language processing (Devlin et al., 2018; He et al., 2016; Krizhevsky et al., 2012; Lan et al., 2019; Mikolov et al., 2013), data-driven meth-ods evade our dream of building a system with basic, every-day, mathematical skills.”
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “E.g., deeper ResNets performed better (He et al., 2016b), bigger Transformer models achieved better translation quality (Vaswani et al., 2017), models with larger vocabulary, or embedding or feature crosses work better, too (Arivazhagan et al., 2019; Conneau et al., 2019).”
- Mixture-of-Experts with Expert Choice Routing (2022, background): “Scaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27].”
- Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “Upstream Pre-Training All of our BiT models use a vanilla ResNet-v2 architecture [16], except that we replace all Batch Normalization [21] layers with Group Normalization [60] and use Weight Standardization [43] in all convolutional layers.”
- Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “Architectures and training details The frame encoder f is modeled using the ResNet-50 v2 [23] architecture with BatchNorm [27].”
- A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, methodology): “In linear/MLP evaluation, widely used in for self-supervised representations (Doersch et al., 2015; Zhang et al., 2016; Noroozi & Favaro, 2016; Doersch & Zisserman, 2017), the weights of a pre-trained network are frozen, and a linear layer/MLP is trained on top to predict the labels.”
- Fixing the train-test resolution discrepancy (2019, neutral): “This concurs with prior findings in the literature [14].”
- Learning Deep Transformer Models for Machine Translation (2019, methodology): “For pre-norm Transformer, we define G ( · ) Figure 2: Connection weights for 3-layer encoder: (a) residual connection (He et al., 2016a), (b) dense residual connection (Britz et al., 2017; Dou et al., 2018), (c) multi-layer representation fusion (Wang et al., 2018b)/transparent attention (Bapna et…”
- Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “Our model uses an encoder based on pre-activation ResNet [He et al., 2016a,b], with changes to make it suitable for DIM.”