Data-free Parameter Pruning for Deep Neural Networks

By Suraj Srinivas, R. Venkatesh Babu. 2015

580 citations recorded by the source corpus.

Deep Neural nets (NNs) with millions of parameters are at the heart of many state-of-the-art computer vision systems today. However, recent works have shown that much smaller models can achieve similar levels of performance. In this work, we address the problem of pruning parameters in a trained NN model. Instead of removing individual weights one at a time as done in previous works, we remove one neuron at a time. We show how similar neurons are redundant, and propose a systematic way to remove them. Our experiments in pruning the densely connected layers show that we can remove upto 85\% of the total parameters in an MNIST-trained network, and about 35\% for AlexNet without significantly affecting performance. Our method can be applied on top of most networks with a fully connected layer to give a smaller network.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Some works prune neurons/channels individually based on some importance score [He et al., 2014, Li et al., 2017, Liebenwein et al., 2020, Mussay et al., 2020, 2021, Molchanov et al., 2017, Srinivas and Babu, 2015].”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, methodology): “Other proposed pruning heuristics include pruning based on activations (Hu et al., 2016), redundancy (Mariet & Sra, 2016; Srinivas & Babu, 2015a), per-layer second derivatives (Dong et al., 2017), and energy/computation efficiency (Yang et al., 2017) (e.g., pruning convolutional filters (Li et al.,…”
Pruning Filters for Efficient ConvNets (2016, background): “There has been a signiﬁcant amount of work on reducing the storage and computation costs by model compression (Le Cun et al. (1989); Hassibi & Stork (1993); Srinivas & Babu (2015); Han et al. (2015); Mariet & Sra (2016)).”
Quantized Convolutional Neural Networks for Mobile Devices (2015, background): “On the other hand, parameter compression in fully-connected layers is explored in [3, 7, 11, 30, 2, 12, 28].”
Paraphrasing Complex Network: Network Compression via Factor Transfer (2018, background): “First, network pruning is a way to reduce network complexity by pruning the redundant and non-informative weights in a pretrained model [26, 17, 7].”
Recent advances in convolutional neural networks (2015, neutral): “pruning, there are also works that investigate coarse-grained pruning. Hu et al . [184] propose removing lters that frequently generate zero output activations on the validation set. Srinivas et al . [185] merge similar lters into one, while Mariet et al . [186] merge lters with similar output activ”
Learning to Prune Filters in Convolutional Neural Networks (2018, background): “[1, 35, 28] also proposed to remove the neurons in networks to increase the sparsity.”
Channel Pruning for Accelerating Very Deep Neural Networks (2017, background): “Some works [45, 35, 19] focus on model size compression, which mainly operate the fully connected layers.”
Rethinking the Value of Network Pruning (2018, background): “Srinivas & Babu (2015) proposes a data-free algorithm to remove redundant neurons iteratively.”
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications (2018, background): “[20] focus on removing entire filters instead of individual weights.”
Diversity Networks (2015, neutral): “In (Srinivas and Babu, 2015), a neuron is pruned when its weights are similar to those of another neuron in its layer, leading to a weight update procedure that is somewhat similar in idea (albeit simpler) to our reweighting step: where (Srinivas and Babu, 2015) removes neurons with equal or similar”
N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning (2017, methodology): “…Pruning-based methods preserve the weights that matter most and remove the redundant weights (LeCun et al. 1989), (Hassibi, Stork, and Wolff 1993), (Srinivas and Babu 2015), (Han et al. 2015), (Han, Mao, and Dally 2015), (Mariet and Sra 2015), (Anwar, Hwang, and Sung 2015), (Guo, Yao, and Chen…”
NISP: Pruning Networks Using Neuron Importance Score Propagation (2017, background): “[32] pruned neurons that are similar to each other.”
Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning (2016, neutral): “[13] proposed merging similar filters into one.”
Data-Free Learning of Student Networks (2019, background): “Lopes et al. [16] utilized the “meta-data” (e.g. means and standard deviation of activations from each layer) recorded from the original training dataset, which is not provided for most well-trained CNNs. Srinivas and Babu [26] compressed the pre-trained network by merging similar neurons in fullyco”
Deep Rewiring: Training very sparse deep networks (2017, methodology): “There are however several heuristic methods for pruning a larger network (Han et al., 2015b;a; Collins & Kohli, 2014; Yang et al., 2015; Srinivas & Babu, 2015), that is, the network is ﬁrst trained to convergence, and network connections and / or neurons are pruned only subsequently.”
Compression of Neural Machine Translation Models via Pruning (2016, methodology): “Other approaches focus on pruning neurons rather than parameters, via sparsity-inducing regu-larizers (Murray and Chiang, 2015) or ‘wiring to-gether’ pairs of neurons with similar input weights (Srinivas and Babu, 2015).”