Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
By Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, Alexander A. Alemi. 2016
15,641 citations recorded by the source corpus.
Very deep convolutional networks have been central to the largest advances in image recognition performance in recent years. One example is the Inception architecture that has been shown to achieve very good performance at relatively low computational cost. Recently, the introduction of residual connections in conjunction with a more traditional architecture has yielded state-of-the-art performance in the 2015 ILSVRC challenge; its performance was similar to the latest generation Inception-v3 network. This raises the question: Are there any benefits to combining Inception architectures with residual connections? Here we give clear empirical evidence that training with residual connections accelerates the training of Inception networks significantly. There is also some evidence of residual Inception networks outperforming similarly expensive Inception networks without residual connections by a thin margin. We also present several new streamlined architectures for both residual and non-residual Inception networks. These variations improve the single-frame recognition performance on the ILSVRC 2012 classification task significantly. We further demonstrate how proper activation scaling stabilizes the training of very wide residual Inception networks. With an ensemble of three residual and one Inception-v4 networks, we achieve 3.08% top-5 error on the test set of the ImageNet classification (CLS) challenge.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Dacheng Tao
- Sergey Levine
- Trevor Darrell
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Trevor Darrell
- Yi Yang
- Jitendra Malik
- Andrew Zisserman
In Model Architectures
- Quoc V. Le
- Alan L. Yuille
- Yoshua Bengio
- Jiashi Feng
- Christoph Feichtenhofer
- Kaiming He
- Dacheng Tao
- Shuicheng Yan
Citations in the corpus
- Xception: Deep Learning with Depthwise Separable Convolutions (2016, methodology): “in 2014 [20] as GoogLeNet (Inception V1), later refined as Inception V2 [7], Inception V3 [21], and most recently Inception-ResNet [19].”
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Additionally, the top-5 accuracy of CLIP models are noticeably higher than their top-1, and this model has a 95% top-5 accuracy, matching Inception-V4 (Szegedy et al., 2016).”
- Mask R-CNN (2017, neutral): “2 Faster R-CNN by G-RMI [21] Inception-ResNet-v2 [37] 34.”
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “There is empirical (Szegedy et al., 2017) and theoretical (Li et al., 2019) evidence showing that a combination of batch normalization and dropout in Convolutional Neural Networks may have harmful results.”
- Do Better ImageNet Models Transfer Better? (2018, methodology): “, 448 × 448 [40] and 768× 768 [9]) than the fine-tuned networks examined in this work.”
- Exploring the Limits of Weakly Supervised Pretraining (2018, neutral): “We believe our results will generalize to other architectures [24,25,26].”
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “2x 13B 13x Inception-resnet-v2 (Szegedy et al., 2017) 80.”
- Wide Residual Networks (2016, background): “Also, follow-up work showed that residual links speed up convergence of deep networks [28].”
- Don't Decay the Learning Rate, Increase the Batch Size (2017, methodology): “Combining these strategies, we train Inception-ResNet-V2 on ImageNet to 77 % validation accuracy in under 2500 parameter updates, using batches of 65536 images.”
- Self-Training With Noisy Student Improves ImageNet Classification (2019, neutral): “0% Inception-resnet-v2 [79] 56M 80.”
- Fixing the train-test resolution discrepancy (2019, methodology): “We used our method with two types of networks for transfer learning tasks: SENet-154 [3] and InceptionResNet-V2 [36].”
- Attention Augmented Convolutional Networks (2019, background): “These observations have inspired the research and design of new architectures, which are typically derived from the composition of convolution operations across an array of spatial scales and skip connections [23, 41, 39, 40, 14, 47, 13].”
- Group Normalization (2018, background): “As a result, many recent models [2,3,5,6,7] are trained with non-trivial batch sizes that are memory-consuming.”
- Revisiting Unreasonable Effectiveness of Data in Deep Learning Era (2017, methodology): “ checkpoints, and keep them fixed throughout for fairer comparison. DurMethod mAP@0.5 mAP@[0.5,0.95] He et al. [16] 53.3 32.2 ImageNet 53.6 34.3 300M 56.9 36.7 ImageNet+300M 58.0 37.4 Inception ResNet [38] 56.3 35.5 Table 2. Object detection performance comparisons with baseline methods on the COCO t”
- Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “The pipeline trains a model with an Inception-V4 backbone (Szegedy et al., 2017) by first pretraining on ImageNet, and then fine-tuning using de-identified retrospective fundus images from EyePACS in the”
- Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, neutral): “inceptionv4 [87].”
- Do ImageNet Classifiers Generalize to ImageNet? (2019, background): “[53] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A.”
- Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, methodology): “All models in this paper use Inception-ResNet-v2 as the CNN component (Szegedy et al., 2016).”
- VOLO: Vision Outlooker for Visual Recognition (2021, background): “Inceptions [48, 49, 47] and ResNeXt [65] examine the design principles of the model building blocks and introduce multiple parallel paths of sets of specialized filters.”