Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
By Kaiming He, X. Zhang, Shaoqing Ren. 2015
20,740 citations recorded by the source corpus.
Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work, we study rectifier neural networks for image classification from two aspects. First, we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit. PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities. This method enables us to train extremely deep rectified models directly from scratch and to investigate deeper or wider network architectures. Based on the learnable activation and advanced initialization, we achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset. This is a 26% relative improvement over the ILSVRC 2014 winner (GoogLeNet, 6.66% [33]). To our knowledge, our result is the first to surpass the reported human-level performance (5.1%, [26]) on this dataset.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Sergey Levine
- Percy Liang
- Dacheng Tao
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
Citations in the corpus
- Deep Residual Learning for Image Recognition (2015, methodology): “1 PReLU-net [13] 21.”
- Convolutional Sequence to Sequence Learning (2017, methodology): “Models with many layers often rely on shortcut or residual connections (He et al., 2015a; Zhou et al., 2016; Wu et al., 2016).”
- Rethinking the Inception Architecture for Computer Vision (2015, methodology): “The computational cost of Inception is also much lower than VGGNet or its higher performing successors [6].”
- Highway Networks (2015, methodology): “For plain deep networks, training with SGD stalls at the beginning unless a specific weight initialization scheme is used such that the variance of the signals during forward and backward propagation is preserved initially (Glorot & Bengio, 2010; He et al., 2015).”
- Object Detection Networks on Convolutional Feature Maps (2015, methodology): “When the NoC weights are randomly initialized, we use the scaled Gaussian distribution [14].”
- Learning Transferable Visual Models From Natural Language Supervision (2021, background): “In 2015, it was announced that a deep learning model exceeded human performance on the ImageNet test set (He et al., 2015).”
- U-Net: Convolutional Networks for Biomedical Image Segmentation (2015, background): “2/N , where N denotes the number of incoming nodes of one neuron [5].”
- A Simple Method for Commonsense Reasoning (2018, background): “, [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]), their dependence on supervised learning has been challenged as a significant weakness.”
- Rethinking ImageNet Pre-Training (2018, methodology): “Successful forms of normalization include normalized parameter initialization [11, 16] and activation normalization layers [20, 2, 46, 48].”
- Wide Residual Networks (2016, methodology): “Various techniques were suggested to enable training of deeper neural networks, such as well-designed initialization strategies [1, 10], better optimizers [27], skip connections [17, 21], knowledge transfer [4, 22] and layer-wise training [23].”
- Shortcut learning in deep neural networks (2020, background): “In an ever-increasing pace, DNNs were reported as having achieved human-level object classification performance [1], beating world-class human Go, Poker, and Starcraft players [2, 3], detecting cancer from X-ray scans [4], translating text across languages [5], helping combat climate change [6], and”
- Weight Standardization (2019, methodology): “Data normalization is widely used for speed up training through proper initialization based on the assumption of the data distribution [7, 11].”
- Group Normalization (2018, methodology): “To normalize hidden features, initialization methods (LeCun et al. 1998; Glorot and Bengio 2010; He et al. 2015) have been derived based on strong assumptions of feature distributions, which can become invalid when training evolves.”
- Non-local Neural Networks (2017, methodology): “We adopt the method in [20] to initialize the weight layers introduced in the non-local blocks.”
- Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “[85] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.”