SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size
By Forrest N. Iandola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William (Bill) J. Dally, Kurt Keutzer. 2016
8,474 citations recorded by the source corpus.
Recent research on deep neural networks has focused primarily on improving accuracy. For a given accuracy level, it is typically possible to identify multiple DNN architectures that achieve that accuracy level. With equivalent accuracy, smaller DNN architectures offer at least three advantages: (1) Smaller DNNs require less communication across servers during distributed training. (2) Smaller DNNs require less bandwidth to export a new model from the cloud to an autonomous car. (3) Smaller DNNs are more feasible to deploy on FPGAs and other hardware with limited memory. To provide all of these advantages, we propose a small DNN architecture called SqueezeNet. SqueezeNet achieves AlexNet-level accuracy on ImageNet with 50x fewer parameters. Additionally, with model compression techniques we are able to compress SqueezeNet to less than 0.5MB (510x smaller than AlexNet). The SqueezeNet architecture is available for download here: this https URL
Subfields: Artificial Intelligence, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Dacheng Tao
- Percy Liang
- Trevor Darrell
- Sergey Levine
In Efficiency & Compression
Citations in the corpus
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, background): “As mobile phones become ubiquitous, it is also common to hand-craft efficient mobile-size ConvNets, such as SqueezeNets (Iandola et al., 2016; Gholami et al., 2018), MobileNets (Howard et al., 2017; Sandler et al., 2018), and ShuffleNets (Zhang et al., 2018; Ma et al., 2018).”
- Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, background): “Early work [22, 23] achieves this by carefully choosing convolutional layers with smaller kernel and channel size, the following work [24–29] explores new variations of convolution operators.”
- Do ImageNet Classifiers Generalize to ImageNet? (2019, background): “squeezenet1_0 [28] https://github.”
- Measuring the Algorithmic Efficiency of Neural Networks (2020, methodology): “…of the architectures shown in Figure 3 [Szegedy et al., 2014, Simonyan & Zisserman, 2014, He et al., 2015, Xie et al., 2016, Huang et al., 2016, Iandola et al., 2016, Zagoruyko & Komodakis, 2016,Zhang et al., 2017,Howard et al., 2017,Sandler et al., 2018,Ma et al., 2018,Tan & Le, 2019] we used…”
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (2019, background): “Efforts in this area could be broadly categorized as follows: (i) new architectures that are compact by design (Iandola et al. 2016; Howard et al. 2017); (ii) automated neural architecture search (NAS) with reward function set as latency or model size (Wang et al.”
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “Squeezenet (Iandola et al., 2016) and MobileNets (Howard et al., 2017) are specifically engineered image-recognition networks that are an order of magnitude smaller than standard architectures.”
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, methodology): “Replacing 3 × 3 convolu-tions with 1 × 1 convolutions is used in [28] to create a very compact neural network that can achieve ∼ 50 × reduction in the number of parameters while obtaining high accuracy.”
- Pruning Filters for Efficient ConvNets (2016, background): “They also demonstrate that the convolutional layers can be compressed and accelerated (Iandola et al. (2016)), but additionally require sparse ∗Work done at NEC Labs †Supported in part by the NSF under Grant IIS-13-2079 ar X iv :1 60 8.”
- MnasNet: Platform-Aware Neural Architecture Search for Mobile (2018, background): “Another common approach is to directly hand-craft more efficient mobile architectures: SqueezeNet [15] reduces the number of parameters and computation by using lower-cost 1x1 convolutions and reducing filter sizes; MobileNet [11] extensively employs depthwise separable convolution to minimize…”
- FractalNet: Ultra-Deep Neural Networks without Residuals (2016, methodology): “Net with stochastic depth, suggesting that FractalNet may be less prone to overfitting. Most methods perform similarly on SVHN. Increasing depth to 40, while borrowing some parameter reduction tricks (Iandola et al., 2016), reveals FractalNet’s performance to be consistent across a range of configurat”
- BAM: Bottleneck Attention Module (2018, methodology): “To demonstrate the advantage in more practical settings, we incorporate our module with compact networks [18, 22], which have tight resource constraints.”
- Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution (2018, background): “Recently, several architectures have been manually designed that aim at reducing resource-consumption while retaining high predictive performance (Iandola et al., 2016; Howard et al., 2017; Sandler et al., 2018).”
- Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, neutral): “Our work also relates to efficient deep learning on edge devices, such as model compression [43, 44, 45], manually designed architectures (MobileNets [46], ShuffeNets [47], SqueezeNets [48]), or even efficient neural architecture search (EfficientNets [4], FBNet [49]).”
- Paraphrasing Complex Network: Network Compression via Factor Transfer (2018, methodology): “There are many ways to directly design efficient small networks such as SqueezeNet (Iandola et al., 2016), MobileNet (Howard et al.”