Memory Bounded Deep Convolutional Networks

By Maxwell D. Collins, Pushmeet Kohli. 2014

174 citations recorded by the source corpus.

In this work, we investigate the use of sparsity-inducing regularizers during training of Convolution Neural Networks (CNNs). These regularizers encourage that fewer connections in the convolution and fully connected layers take non-zero values and in effect result in sparse connectivity between hidden units in the deep network. This in turn reduces the memory and runtime cost involved in deploying the learned CNNs. We show that training with such regularization can still be performed using stochastic gradient descent implying that it can be used easily in existing codebases. Experimental evaluation of our approach on MNIST, CIFAR, and ImageNet datasets shows that our regularizers can result in dramatic reductions in memory requirements. For instance, when applied on AlexNet, our method can reduce the memory consumption by a factor of four with minimal loss in accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, methodology): “Many variants have been proposed (Collins & Kohli, 2014; Han et al., 2015; Guo et al., 2016; Zhu & Gupta, 2017), with the key differences lying in when weights are removed, whether weights should be sorted to remove a precise proportion or thresholded based on a ﬁxed or decaying value, and whether…”
Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Other methods employ sparsity and group-sparsity regularisation during training, to induce sparse weights [Collins and Kohli, 2014, Voita et al., 2019].”
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015, background): “(Collins & Kohli, 2014) reduced the parameters by 4 × and with inferior accuracy.”
MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (2017, background): “, each individual parameter is equally valuable) [3, 19, 32].”
Data-free Parameter Pruning for Deep Neural Networks (2015, background): “Collins and Kohli [5] propose a sparsity inducing regulariser for backpropogation which promotes many weights to have zero magnitude.”
Fast ConvNets Using Group-Wise Brain Damage (2015, background): “Similarly to [32], the work [14] uses sparsification to reduce the number of parameters in the memory-bound scenario.”
Accelerating Very Deep Convolutional Networks for Classification and Detection (2015, methodology): “We first propose a response reconstruction method that takes into account the nonlinear neurons and a low-rank constraint.”
Deep Fried Convnets (2014, methodology): “The result of [8] is based on the the Caffe AlexNet model (similar but not identical to the Caffe reference model) and achieves ∼4x reduction in memory usage, (slightly better than Fastfood 16 but with a noted drop in performance).”
Learning the Number of Neurons in Deep Networks (2016, methodology): “me of the neurons, exploiting standard regularizers on the individual parameters has proven effective in the past for generalization purpose [Bartlett, 1996, Krogh and Hertz, 1992, Theodoridis, 2015, Collins and Kohli, 2014]. To further leverage this idea within our automatic model selection approac”
Deep Rewiring: Training very sparse deep networks (2017, methodology): “There are however several heuristic methods for pruning a larger network (Han et al., 2015b;a; Collins & Kohli, 2014; Yang et al., 2015; Srinivas & Babu, 2015), that is, the network is ﬁrst trained to convergence, and network connections and / or neurons are pruned only subsequently.”
PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions (2015, background): “Several papers have demonstrated that it is possible to compress the parameters of the fully-connected layers (where most CNN parameters reside) with a marginal error increase [4, 21, 27].”
Structured Pruning of Deep Convolutional Neural Networks (2015, background): “Computational complexity is reduced with sparse connectivity in convolution and fully connected layers in [10].”
Sparsifying Neural Network Connections for Face Recognition (2015, methodology): “We compare our neural correlation based weight pruning strategy with other pruning strategies proposed previously, including optimal brain damage (OBD) [7], weight magnitude based pruning (which is the bases of many weight pruning algorithms [6]), and Bayesian regularization and pruning (BRP) [27].”
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021, background): “Many variants have been proposed (Collins and Kohli, 2014; Han et al., 2015; Guo et al., 2016; Zhu and Gupta, 2017), which can be distinguished by differences in the criteria used to remove weights, when weights are removed and whether weights that have been pruned can still receive gradient…”
Less Is More: Towards Compact CNNs (2016, methodology): “In [26], directly adding sparse constraints on the parameters was discussed.”
Nonparametric Neural Networks (2017, background): “(For a basic comparison of 1-norm and 2-norm regularizers, see Yuan & Lin (2006) and for a comparison in the context of neural networks, see Collins & Kohli (2014).)”
Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype (2018, methodology): “This neural network architecture was previously used to benchmark the accuracy of sparsely connected low-memory footprint networks (Collins and Kohli, 2014; Han et al., 2015).”