ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
By Ningning Ma, Xiangyu Zhang, Haitao Zheng. 2018
6,472 citations recorded by the source corpus.
Currently, the neural network architecture design is mostly guided by the indirect metric of computation complexity, i.e., FLOPs. However, the direct metric, e.g., speed, also depends on the other factors such as memory access cost and platform characterics. Thus, this work proposes to evaluate the direct metric on the target platform, beyond only considering FLOPs. Based on a series of controlled experiments, this work derives several practical guidelines for efficient network design. Accordingly, a new architecture is presented, called ShuffleNet V2. Comprehensive ablation experiments verify that our model is the state-of-the-art in terms of speed and accuracy tradeoff.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Dacheng Tao
- Jianfeng Gao
- Mohit Bansal
- Yoshua Bengio
- Trevor Darrell
- Jiashi Feng
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Chunhua Shen
- Alan L. Yuille
- Yi Yang
In Neurology
In Efficiency & Compression
Citations in the corpus
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, background): “As mobile phones become ubiquitous, it is also common to hand-craft efficient mobile-size ConvNets, such as SqueezeNets (Iandola et al., 2016; Gholami et al., 2018), MobileNets (Howard et al., 2017; Sandler et al., 2018), and ShuffleNets (Zhang et al., 2018; Ma et al., 2018).”
- Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, neutral): “Early work in this direction includes [23, 31, 13, 17, 32, 16, 48, 27, 43].”
- Local Relation Networks for Image Recognition (2019, background): “Impact of spatial scope Table 3 presents the impact of varying aggregation spatial scope for the proposed local relation networks, as well as the regular ResNet-26 network and its variant, ResNet-DW-26 [21], where the regular convolution layer is replaced by depthwise convolution.”
- Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, neutral): “shufflenet_v2_x0_5 [54].”
- Bag of Tricks for Image Classification with Convolutional Neural Networks (2018, methodology): “This tweak was proposed in Inception-v2 [26] originally, and it can be found on the implementations of other models, such as SENet [12], PSPNet [31], DeepLabV3 [1], and ShuffleNetV2 [21].”
- Measuring the Algorithmic Efficiency of Neural Networks (2020, methodology): “…2014, He et al., 2015, Xie et al., 2016, Huang et al., 2016, Iandola et al., 2016, Zagoruyko & Komodakis, 2016,Zhang et al., 2017,Howard et al., 2017,Sandler et al., 2018,Ma et al., 2018,Tan & Le, 2019] we used PyTorch’s example models [Paszke et al., 2017] with Pytorch’s suggested hyperparameters.”
- Per-Pixel Classification is Not All You Need for Semantic Segmentation (2021, neutral): “8 Auto-Panoptic [48] ShuffleNetV2 [32] 32.”
- The Efficiency Misnomer (2021, methodology): “FNet (Lee-Thorp et al., 2021) also shows how certain architectures can have significantly different speed in different hardware (e.g., GPUs and TPUs).”
- Designing Network Design Spaces (2020, background): “We observe that REGNETS are surprisingly effective in this regime considering the substantial body of work on finding better mobile networks via both manual design [7, 21, 16] and NAS [30, 19, 14, 15].”
- Green AI (2019, background): “For example, a significant amount of work in the computer vision community has addressed efficient inference, which is necessary for real-time processing of images for applications like self-driving cars [23, 30, 22], or for placing models on devices such as mobile phones [16, 33].”
- MnasNet: Platform-Aware Neural Architecture Search for Mobile (2018, background): “…and reducing filter sizes; MobileNet [11] extensively employs depthwise separable convolution to minimize computation density; ShuffleNets [33, 24] utilize low-cost group convolution and channel shuffle; Con-densenet [14] learns to connect group convolutions across layers; Recently,…”
- Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, background): “Batch Normalization (BN) (Ioffe & Szegedy, 2015) is the current workhorse in convolutional deep learning tasks and has been employed by default in most SOTA CNNs (He et al., 2016; Huang et al., 2017; Sandler et al., 2018; Ma et al., 2018; Tan & Le, 2019), however often fails on heterogeneous trainin”
- SNAS: Stochastic Neural Architecture Search (2018, background): “Rather than directly estimating the forwarding time, there are three candidates from the literature (Gordon et al., 2018; Ma et al., 2018) that can be used to approximately represent it: 1) the parameter size ; 2) the number of float-point operations (FLOPs); and 3) the memory access cost (MAC).”
- Similarity-Preserving Knowledge Distillation (2019, methodology): “The ShuffleNetV2 networks are summarized in Table 3 in the supplementary.”
- AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, positive): “We did not include dilated convolutions, motivated by their hardware unfriendliness as analyzed in (Ma et al., 2018).”
- FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, methodology): “This includes convolutional variants like depthwise convolutions in MobileNet, inverted residual blocks in MobileNetV2; activations such as hswish in MobileNetV3 [14, 31, 13]; and memory-based operations like shift [39] and shuffle [24].”
- GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (2020, methodology): “Assume the supernet N has L layers, and each layer N l is allocated with O operation choices O = {oi}, which can be basic convolution, pooling, identity or different types of building blocks, such as MobileNetV2 block [28] and ShuffleNetV2 block [23].”
- BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, methodology): “We also tried many other learning rate schedules with an exhaustive hyper-parameter sweep, including linearly decaying [24, 36] and cosine decaying [13, 23].”
- Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “The second trend is to design a tricky lightweight model to provide satisfying accuracy with fewer weights and parameters (Ma et al., 2018; Sandler et al., 2018; Tan et al., 2019).”
- DSNAS: Direct Neural Architecture Search Without Parameter Retraining (2020, methodology): “Search Space The basic building block design is inspired by ShuffleNet v2 [15].”