Deep Pyramidal Residual Networks

By Dongyoon Han, Jiwhan Kim, Junmo Kim. 2016

759 citations recorded by the source corpus.

Deep convolutional neural networks (DCNNs) have shown remarkable performance in image classification tasks in recent years. Generally, deep neural network architectures are stacks consisting of a large number of convolutional layers, and they perform downsampling along the spatial dimension via pooling to reduce memory usage. Concurrently, the feature map dimension (i.e., the number of channels) is sharply increased at downsampling locations, which is essential to ensure effective performance because it increases the diversity of high-level attributes. This also applies to residual networks and is very closely related to their performance. In this research, instead of sharply increasing the feature map dimension at units that perform downsampling, we gradually increase the feature map dimension at all units to involve as many locations as possible. This design, which is discussed in depth together with our new insights, has proven to be an effective means of improving generalization ability. Furthermore, we propose a novel residual unit capable of further improving the classification accuracy with our new network architecture. Experiments on benchmark CIFAR-10, CIFAR-100, and ImageNet datasets have shown that our network architecture has superior generalization ability compared to the original residual networks. Code is available at https://github.com/jhkim89/PyramidNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Do ImageNet Classifiers Generalize to ImageNet? (2019, background): “[19] Dongyoon Han, Jiwhan Kim, and Junmo Kim.”
SGDR: Stochastic Gradient Descent with Warm Restarts (2016, background): “Alternative network structures should be also considered; e.g., soon after our initial arXiv report (Loshchilov & Hutter, 2016), Zhang et al. (2016); Huang et al. (2016b); Han et al. (2016) reported that WRNs models can be replaced by more memory-efﬁcient models.”
Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “Architectures presented include PyramidNet [15], Wide-ResNet-28-10 [53], ResNet-50 [17], and EfﬁcientNet-B7 [47].”
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019, methodology): “We use PyramidNet-200 trained on CIFAR-100 datasets with same setting as in Section 4.1.2.”
Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, methodology): “We first evaluate SAM’s impact on generalization for today’s state-of-the-art models on CIFAR-10 and CIFAR-100 (without pretraining): WideResNets with ShakeShake regularization (Zagoruyko and Komodakis, 2016; Gastaldi, 2017) and PyramidNet with ShakeDrop regularization (Han et al., 2016; Yamada et a”
Squeeze-and-Excitation Networks (2017, neutral): “PyramidNet-200 [8] 20.”
MobileNetV2: Inverted Residuals and Linear Bottlenecks (2018, neutral): “We note that similar reports where non-linearity was helped were reported in [29] where non-linearity was removed from the input of the traditional residual block and that lead to improved performance on CIFAR dataset.”
Born Again Neural Networks (2018, background): “Multiple extensions [17, 16, 12, 19] have been proposed, progressively increasing their accuracy on CIFAR100 [20] and Imagenet [21].”
Regularized Evolution for Image Classifier Architecture Search (2018, background): “Recent neural network successes have encouraged a proliferation of model architectures (He et al. (2016); Huang et al.”
Do CIFAR-10 Classifiers Generalize to CIFAR-10? (2018, methodology): “…all gradients by 5 • pyramidnet_basic_110_270 : PyramidNet [6] with depth 110, block type “basic", α = 270 • pyramidnet_basic_110_84 : PyramidNet [6] with depth 110, block type “basic", α = 84 • random_features_256k_aug [2]: Random 1 layer convolutional network with 256k ﬁlters sampled from…”
BAM: Bottleneck Attention Module (2018, background): “Similarly, PyramidNet [13] shows that increasing channels in deeper layers can effectively boost the performance.”
Averaging Weights Leads to Wider Optima and Better Generalization (2018, methodology): “Additionally, we experiment with the recent Shake-Shake2x64d [Gastaldi, 2017] on CIFAR-10 and PyramidNet272 (bottleneck, α = 200) [Han et al., 2016] on CIFAR100.”
Real-Time Federated Evolutionary Neural Architecture Search (2020, positive): “The intuition of using expanding at the ﬁrst layer instead of in the last layer as done in the bottleneck layer is that a nonlinear activation function such as ReLu may cause layer information loss [49] and using a nonlinear projection upon a high-dimensional space can mitigate this issue.”
Adversarial AutoAugment (2019, neutral): “We evaluate our proposed method with the following models: Wide-ResNet-2810 (Zagoruyko & Komodakis, 2016), Shake-Shake (26 2x32d) (Gastaldi, 2017), Shake-Shake (26 2x96d) (Gastaldi, 2017), Shake-Shake (26 2x112d) (Gastaldi, 2017), PyramidNet+ShakeDrop (Han et al., 2017; Yamada et al., 2018).”
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules (2019, methodology): “…as random cropping, ﬂipping, rotating, scaling, and translating are used by top performing models for training on MINST, CIFAR-10, and ImageNet datasets (Simard et al., 2003; Ciresan et al., 2012; Wan et al., 2013; Sato et al., 2015; Krizhevsky et al., 2012; Lin et al., 2013; Han et al., 2016).”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, background): “Params Search Cost Search Methodtop-1 top-5 (M) (GPU days) Inception-v1 (Szegedy et al., 2015) 30.2 10.1 6.6 – manual MobileNet (Howard et al., 2017) 29.4 10.5 4.2 – manual ShuffleNet 2× (v1) (Zhang et al., 2018) 29.1 10.2 ∼5 – manual ShuffleNet 2× (v2) (Zhang et al., 2018) 26.3 – ∼5 – manual NASNet”
EENA: Efficient Evolution of Neural Architecture (2019, methodology): “Human-designed neural architectures such as ResNet [14], DenseNet [17], PyramidNet [13] and so on which contain several effective blocks are successively proposed to increase the accuracy of image classification.”
Fast AutoAugment (2019, methodology): “On the benchmark image dataset, such as CIFAR and ImageNet, random crop, flip, rotation, scaling, and color transformation, have been performed as baseline augmentation methods [10, 21, 30].”