Less Is More: Towards Compact CNNs

By Hao Zhou, José Manuel Álvarez, Fatih Murat Porikli. 2016

340 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neurology

Citations in the corpus

Learning to Prune Filters in Convolutional Neural Networks (2018, background): “[1, 35, 28] also proposed to remove the neurons in networks to increase the sparsity.”
MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (2017, background): “, each individual parameter is equally valuable) [3, 19, 32].”
Learning Efficient Convolutional Networks through Network Slimming (2017, background): “[37] imposes neuron-level sparsity during training thus some neurons could be pruned to obtain compact networks.”
Channel Pruning for Accelerating Very Deep Neural Networks (2017, background): “[54] could work well for fully connected layers.”
Rethinking the Value of Network Pruning (2018, methodology): “Group sparsity is also widely used to smooth the pruning process after training (Wen et al., 2016; Alvarez & Salz-mann, 2016; Lebedev & Lempitsky, 2016; Zhou et al., 2016).”
Learning the Number of Neurons in Deep Networks (2016, background): “However, [Zhou et al., 2016] focuses on the last fully-connected layer to obtain a compact model, and [S. Tu, 2014] and [Murray and Chiang, 2015] only considered small networks.”
Data-Driven Sparse Structure Selection for Deep Neural Networks (2017, methodology): “[45, 1] added group sparsity regularizations on the weights of neurons and sparsified them in the training stage.”
More is Less: A More Complicated Network with Less Inference Complexity (2017, background): “In [38], many neurons were decimated by incorporating sparse constraints into the objective function.”
Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers (2018, background): “Some recent efforts proposed to impose structured sparsity on those vector components motivated from the implementation perspective on specialized hardware (Wen et al., 2016; Zhou et al., 2016; Alvarez & Salzmann, 2016; Lebedev & Lempitsky, 2016).”
Domain-Adaptive Deep Network Compression (2017, methodology): “This process can be conducted while training [2, 38, 14, 20], or by analyzing the inﬂuence of each parameter once the network has been trained [22].”
End-to-End Learning of Energy-Constrained Deep Neural Networks (2018, methodology): “Subsequent work guides pruning using special structures [26, 34, 25, 31], such as removing an entire channel, to better retain accuracy after pruning.”
GAN-Knowledge Distillation for One-Stage Object Detection (2019, background): “Reference [24] imposes neuronlevel sparsity in training thus some neurons could be pruned to obtain compact networks.”
A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers (2018, methodology): “mputation and model storage requirements of DNNs, with minor eect on the overall accuracy. These model compression techniques include weight pruning [9,10,11,12,13,14,15,16], sparsity regularization [17,18,19], weight clustering [9,20,21], and low rank approximation [22,23], etc. A simple but eectiv”
CLIP-Q: Deep Network Compression Learning by In-parallel Pruning-Quantization (2018, background): “This re-Layer -0.3% 68 MB Less Is More [60] -0.6% 57 MB Dynamic Network Surgery [13] -0.2% 1 13.8 MB Circulant CNN [3] -0.4% 12.7 MB Quantized CNN [53] -1.4% 12.6 MB Binary-Weight-Networks [41] +0.1% 7.6 MB Deep Compression [16] +0.0% 8.9 (6.9) MB [16] + Weighted-Entropy Quantization [38] -0.8%”
Compression-aware Training of Deep Networks (2017, methodology): “Other methods have proposed to make use of sparsity-inducing techniques to cancel out individual parameters [49, 10, 20, 19, 35] or units [2, 50, 55].”
Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization (2018, background): “Structural pruning identifies channels and layers to remove from the architecture, thus is able to directly achieve speedup without the need of specialized implementations [1, 25, 35, 41, 45, 59, 68].”