SqueezeNext: Hardware-Aware Neural Network Design

By A. Gholami, Kiseok Kwon, Bichen Wu, Zizheng Tai, Xiangyu Yue, Peter H. Jin, Sicheng Zhao, Kurt Keutzer. 2018

329 citations recorded by the source corpus.

One of the main barriers for deploying neural networks on embedded systems has been large memory and power consumption of existing neural networks. In this work, we introduce SqueezeNext, a new family of neural network architectures whose design was guided by considering previous architectures such as SqueezeNet, as well as by simulation results on a neural network accelerator. This new network is able to match AlexNet's accuracy on the ImageNet benchmark with 112× fewer parameters, and one of its deeper variants is able to achieve VGG-19 accuracy with only 4.4 Million parameters, (31× smaller than VGG-19). SqueezeNext also achieves better top-5 classification accuracy with 1.3× fewer parameters as compared to MobileNet, but avoids using depthwise-separable convolutions that are inefficient on some mobile processor platforms. This wide range of accuracy gives the user the ability to make speed-accuracy tradeoffs, depending on the available resources on the target hardware. Using hardware simulation results for power and inference speed on an embedded system has guided us to design variations of the baseline model that are 2.59×/8.26× faster and 2.25×/7.5× more energy efficient as compared to SqueezeNet/AlexNet without any accuracy degradation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Efficiency & Compression

Citations in the corpus

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, background): “As mobile phones become ubiquitous, it is also common to hand-craft efﬁcient mobile-size ConvNets, such as SqueezeNets (Iandola et al., 2016; Gholami et al., 2018), MobileNets (Howard et al., 2017; Sandler et al., 2018), and ShufﬂeNets (Zhang et al., 2018; Ma et al., 2018).”
Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, background): “Early work [22, 23] achieves this by carefully choosing convolutional layers with smaller kernel and channel size, the following work [24–29] explores new variations of convolution operators.”
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (2019, methodology): “Efforts in this area could be broadly categorized as follows: (i) new architectures that are compact by design [11, 13]; (ii) automated neural architecture search (NAS) with reward function set as latency or model size [32, 35]; (iii) pruning based methods to reduce model size of existing architectu”
MnasNet: Platform-Aware Neural Architecture Search for Mobile (2018, neutral): “amples a batch of models using its current parameters , by Model Type #Params #Mult-Adds Top-1 Acc. (%) Top-5 Acc. (%) Inference Latency MobileNetV1 [11] manual 4.2M 575M 70.6 89.5 113ms SqueezeNext [5] manual 3.2M 708M 67.5 88.2 - ShufﬂeNet (1.5x) [33] manual 3.4M 292M 71.5 - - ShufﬂeNet (2x) manua”
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search (2018, methodology): “Following SqueezeNet, SqueezeNext [3] and ShiftNet [23] achieve further parameter size reduction.”
HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision (2019, methodology): “SqueezeNext [7] is an example work here where the neural network and associated accelerator are co-designed.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “…NN architecture design, initial approaches at architecture-hardware co-design were manual, where an expert would adapt/change the NN architecture [70], followed by using automated AutoML and/or NAS techniques [22, 23, 99, 244]. c) Pruning : Another approach to reducing the memory footprint and…”
I-BERT: Integer-only BERT Quantization (2021, neutral): “These techniques can be broadly categorized into (1) model pruning [18, 31, 35, 38, 40, 67], (2) knowledge distillation [21, 39, 43, 49, 70], (3) efficient neural architecture design [23, 24, 37, 51, 57], (4) hardware and neural architecture co-design [16, 17, 22, 29, 64], and (5) quantization [5, 7”
HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks (2019, methodology): “Also note that [5, 28, 30] We also apply HAWQ-V2 to quantize deep and highly compact models such as SqueezeNext [8].”
Synetgy: Algorithm-hardware Co-design for ConvNet Accelerators on Embedded FPGAs (2018, methodology): “To achieve AlexNet-level accuracy, SqueezeNet [9] is 50x smaller than AlexNet; SqueezeNext [10] is 112x smaller; ShiftNet-C [11], with 1.”
Fast and Practical Neural Architecture Search (2019, unknown): “SqueezeNext [6] manual 3.”
ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs (2019, background): “We discussed results on Cifar-10/100 dataset using variants of Residual and SqueezeNext networks.”
Review of Image Classification Algorithms Based on Convolutional Neural Networks (2021, methodology): “After that, SqueezeNext [160] considered the hardware based on SqueezeNet.”
ZeroQ: A Novel Zero Shot Quantization Framework (2020, methodology): “These meth-ods are typically orthogonal to quantization, and they include efﬁcient neural architecture design [17, 9, 16, 36, 43], knowledge distillation [14, 35], model pruning [11, 29, 24], and hardware and NN co-design [9, 21].”
SqueezeWave: Extremely Lightweight Vocoders for On-device Speech Synthesis (2020, background): “Previous work (Iandola et al., 2016; Wu et al., 2016; 2017; 2018a;b; Yang et al., 2018; Gholami et al., 2018; Wu, 2019; Wu et al., 2019; Dai et al., 2019) have shown that optimizing the neural network architecture can lead to significant efficiency improvement in many applications.”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “Instead of using a ﬁre module, a two-stage bottleneck module is used in SqueezeNext [74].”
HAWQV3: Dyadic Neural Network Quantization (2020, background): “…new NN architectures (Iandola et al., 2016; Sandler et al., 2018; Tan & Le, 2019); (ii) Co-designing NN architecture and hardware together (Gholami et al., 2018; Han & Dally, 2017; Howard et al., 2019; Wu et al., 2019); (iii) Pruning redundant ﬁlters (Han et al., 2015; LeCun et al., 1990;…”