Fast Sparse ConvNets

By Erich Elsen, Erich Elsen, Marat Dukhan, Marat Dukhan, Trevor Gale, Trevor Gale, Karen Simonyan, Karen Simonyan. 2019

161 citations recorded by the source corpus.

Historically, the pursuit of efficient inference has been one of the driving forces behind the research into new deep learning architectures and building blocks. Some of the recent examples include: the squeeze-and-excitation module, depthwise separable convolutions in Xception, and the inverted bottleneck in MobileNet v2. Notably, in all of these cases, the resulting building blocks enabled not only higher efficiency, but also higher accuracy, and found wide adoption in the field. In this work, we further expand the arsenal of efficient building blocks for neural network architectures; but instead of combining standard primitives (such as convolution), we advocate for the replacement of these dense primitives with their sparse counterparts. While the idea of using sparsity to decrease the parameter count is not new, the conventional wisdom is that this reduction in theoretical FLOPs does not translate into real-world efficiency gains. We aim to correct this misconception by introducing a family of efficient sparse kernels for several hardware platforms, which we plan to open source for the benefit of the community. Equipped with our efficient implementation of sparse primitives, we show that sparse versions of MobileNet v1 and MobileNet v2 architectures substantially outperform strong dense baselines on the efficiency-accuracy curve. On Snapdragon 835 our sparse networks outperform their dense equivalents by 1.3 − 2.4× – equivalent to approximately one entire generation of improvement. We hope that our findings will facilitate wider adoption of sparsity as a tool for creating efficient and accurate deep learning architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “, 2018) and convolutional models for computer vision (Zhu & Gupta, 2018; Elsen et al., 2019; Evci et al., 2020; Kusupati et al., 2020).”
MegaBlocks: Efficient Sparse Training with Mixture-of-Experts (2022, background): “Lastly, block-sparse kernels like matrix multiplication and convolution are general-purpose primitives that are useful across a range of applications (Narang et al., 2017; Gray et al., 2017; Child et al., 2019; Elsen et al., 2020).”
The hardware lottery (2020, background): “In parallel, there is interesting research developing specialized software kernels to support un-structured sparsity (Elsen et al., 2020; Gale et al., 2020; Gray et al., 2017).”
Rigging the Lottery: Making All Tickets Winners (2019, background): “Multiple works have shown inference time speedups are possible using sparsity for both Recurrent Neural Networks (RNNs) (Kalchbrenner et al., 2018) and Convolu-tional Neural Networks (ConvNets) (Park et al., 2016; Elsen et al., 2019).”
Sparse GPU Kernels for Deep Learning (2020, background): “[16] design efficient SpMM kernels for CPUs and demonstrate significant performance improvements for highly efficient neural networks on mobile processors.”
Soft Threshold Weight Reparameterization for Learnable Sparsity (2020, background): “Naturally, several interesting projects address inference speed-ups via sparsity on existing frameworks (Liu et al., 2015; Elsen et al., 2019) and commodity hardware (Ashby et al.”
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2023, background): “However, as our base quantization levels are already very low, standard sparsity metadata formats (Elsen et al., 2020; Lin et al., 2023) would only allow limited additional compression.”
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021, background): “The development of specialized software kernels has enabled the acceleration of sparse networks on traditional hardware (Gale et al., 2020; Elsen et al., 2019; Zhu et al., 2019) with new generations of hardware directly facilitating sparse training (Zhu et al., 2019).”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “For compact network architectures, such as MobileNet-v2 and EfﬁcientNet-B0, pruning can introduce 50%–93% sparsity resulting in 2.5 × –4.2 × MAC elimination [223].”
Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture (2020, background): “However, there is active development for possible solutions (Liu et al., 2015; Han et al., 2016; Elsen et al., 2019).”
Layer-adaptive Sparsity for the Magnitude-based Pruning (2020, background): “While such unstructured pruning techniques have been considered less practical (compared to structured pruning), several recent breakthroughs provide promising methods to bridge this gap; see Gale et al. (2020); Elsen et al. (2020).”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, unknown): “Another practically relevant pruning pattern, particularly in the context of CPU acceleration [7, 22], is block-pruning , where zeros appear only in consecutive blocks of size c , which is typically a small number like 4 or 8.”
Playing Lottery Tickets with Vision and Language (2021, background): “For example, XNNPACK [14] has shown significant speedups over dense networks on smartphone processors.”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, background): “Finally, another interesting research direction is the combination of sparsity and quantization, which would allow combining computational speedups from sparsity [26, 5] with memory savings from quantization [9, 3, 4].”
How Well Do Sparse ImageNet Models Transfer? (2021, background): “Thanks to this trend, a set of fairly consistent accuracy benchmarks has emerged for pruning, along with increasingly efﬁcient computational support [11,20,40,52].”
Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better (2021, background): “[52, 66] have found that some architectures like MobileNetV2, EfficientNet [147] have thin first layers that do not contribute significantly to the number of parameters and pruning them leads to an accuracy drop without much gain.”