Recovering from Random Pruning: On the Plasticity of Deep Convolutional Neural Networks

By Deepak Mittal, Shweta Bhardwaj, Mitesh M. Khapra, Balaraman Ravindran. 2018

68 citations recorded by the source corpus.

Recently there has been a lot of work on pruning filters from deep convolutional neural networks (CNNs) with the intention of reducing computations. The key idea is to rank the filters based on a certain criterion (say, l1-norm, average percentage of zeros, etc) and retain only the top ranked filters. Once the low scoring filters are pruned away the remainder of the network is fine tuned and is shown to give performance comparable to the original unpruned network. In this work, we report experiments which suggest that the comparable performance of the pruned network is not due to the specific criterion chosen but due to the inherent plasticity of deep neural networks which allows them to recover from the loss of pruned filters once the rest of the filters are fine-tuned. Specifically, we show counter-intuitive results wherein by randomly pruning 25-50% filters from deep CNNs we are able to obtain the same performance as obtained by using state of the art pruning methods. We empirically validate our claims by doing an exhaustive evaluation with VGG-16 and ResNet-50. Further, we also evaluate a real world scenario where a CNN trained on all 1000 ImageNet classes needs to be tested on only a small set of classes at test time (say, only animals). We create a new benchmark dataset from ImageNet to evaluate such class specific pruning and show that even here a random pruning strategy gives close to state of the art performance. Lastly, unlike existing approaches which mainly focus on the task of image classification, in this work we also report results on object detection. We show that using a simple random pruning strategy we can achieve significant speed up in object detection (74% improvement in fps) while retaining the same accuracy as that of the original Faster RCNN model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Reducing Transformer Depth on Demand with Structured Dropout (2019, background): “Similar to Mittal et al. (2018), we take advantage of the plasticity of neural networks to learn models that are resilient to random pruning, rather than learning the pruning itself.”
Rethinking the Value of Network Pruning (2018, background): “Mittal et al. (2018) shows that random channel pruning (Anwar & Sung, 2016) can perform on par with a variety of more sophisticated pruning criteria, demonstrating the plasticity of network models.”
Training with Quantization Noise for Extreme Model Compression (2020, methodology): “Many compression methods focus on efficient parameterization, via weight pruning [5, 9, 10, 11], weight sharing [12, 13, 14] or with dedicated architectures [4, 15, 16].”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, methodology): “…2 in the main text for two examples, ThiNet and SSS), and emerging evidence has called into question the fundamental value of structured pruning: (Mittal et al., 2018) found that the channel pruning criteria used in a number of state-of-the-art structured pruning methods performed no better than…”
Training with Quantization Noise for Extreme Fixed-Point Compression (2020, methodology): “Many compression methods focus on efficient parameterization, via weight pruning [5, 9, 10, 11], weight sharing [12, 13, 14] or with dedicated architectures [4, 15, 16].”
CAP: Context-Aware Pruning for Semantic Segmentation (2021, background): “Different pruning criteria are regarded as the indicators for redundant filters and have been studied [17, 26, 24], e.”
Efficient Video Classification Using Fewer Frames (2019, background): “Recently, there has been a lot of work on model compression ([7, 26, 12, 25]) in the context of image classification.”
Neural Pruning via Growing Regularization (2020, background): “This is reasonable since when more weights are pruned, the network cannot recover by its inherent plasticity Mittal et al. (2018), then the regularization-based way is more helpful because it helps the model transfer its expressive power to the remaining part.”
Fast Neural Architecture Construction using EnvelopeNets (2018, methodology): “Mittal et. al (27) surveyed several metrics for pruning and showed that random pruning can be as effective as algorithmic pruning when performed on a hand designed network (such as ResNet (16)).”
Neural Architecture Construction using EnvelopeNets (2018, methodology): “al [12] shows that random pruning can be as eective as algorithmic pruning when performed on a hand designed network (such as ResNet).”