Training Sparse Neural Networks

By Suraj Srinivas, Akshayvarun Subramanya, R. Venkatesh Babu. 2016

226 citations recorded by the source corpus.

The emergence of Deep neural networks has seen human-level performance on large scale computer vision tasks such as image classification. However these deep networks typically contain large amount of parameters due to dense matrix multiplications and convolutions. As a result, these architectures are highly memory intensive, making them less suitable for embedded vision applications. Sparse Computations are known to be much more memory efficient. In this work, we train and build neural networks which implicitly use sparse computations. We introduce additional gate variables to perform parameter selection and show that this is equivalent to using a spike-and-slab prior. We experimentally validate our method on both small and large networks which result in highly sparse neural network models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Dynamic Model Pruning with Feedback (2020, background): “Srinivas et al. (2017); Louizos et al. (2018) learn gating variables (e.g. through `0 regularization) that minimize the number of nonzero weights, recent parallel work studies filter pruning for pre-trained models (You et al., 2019).”
Learning Efficient Convolutional Networks through Network Slimming (2017, background): “[32] overcomes this limitation by explicitly imposing sparse constraint over each weight with additional gate variables, and achieve high compression rates by pruning connections with zero gate values.”
Rigging the Lottery: Making All Tickets Winners (2019, background): “The parameter and ﬂoating point operation (FLOP) efﬁ-ciency of sparse neural networks is now well demonstrated on a variety of problems (Han et al., 2015; Srinivas et al., 2017).”
Discrimination-aware Channel Pruning for Deep Neural Networks (2018, background): “Training with sparsity constraints [37, 46] has also been studied to reach higher compression rate.”
Effective Sparsification of Neural Networks with Global Sparsity Constraint (2021, unknown): “[32, 23, 27] achieve strong empirical results while we observe several shortcomings in the reparameterization process.”
Discrimination-Aware Network Pruning for Deep Model Compression (2020, background): “Training with sparsity constraints [67], [78] has also been studied to reach a higher compression rate.”
An end-to-end approach for speeding up neural network inference (2018, methodology): “[46] learns sparsity at the weight level using a binary mask.”
The Lottery Ticket Hypothesis: Training Pruned Neural Networks (2018, background): “training, they randomly reinitialize unimportant ﬁlters. During training. Bellac et al. [4] train with sparse networks and replace weights that reach zero with new random connections. Srinivas et al. [47] and Louizos et al. [35] learn gating variables that minimize the number of nonzero network para”
Channel Selection Using Gumbel Softmax (2020, methodology): “[45] learns sparsity at the weight level using a binary mask.”