Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks

By Andrey Kuzmin, Markus Nagel, Saurabh Pitre, Sandeep Pendyam, Tijmen Blankevoort, Max Welling. 2019

27 citations recorded by the source corpus.

The success of deep neural networks in many real-world applications is leading to new challenges in building more efficient architectures. One effective way of making networks more efficient is neural network compression. We provide an overview of existing neural network compression methods that can be used to make neural networks more efficient by changing the architecture of the network. First, we introduce a new way to categorize all published compression methods, based on the amount of data and compute needed to make the methods work in practice. These are three 'levels of compression solutions'. Second, we provide a taxonomy of tensor factorization based and probabilistic compression methods. Finally, we perform an extensive evaluation of different compression techniques from the literature for models trained on ImageNet. We show that SVD and probabilistic compression or pruning methods are complementary and give the best results of all the considered methods. We also provide practical ways to combine them.

Subfields: Artificial Intelligence, Computational Mathematics, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mathematics

In Efficiency & Compression

Citations in the corpus

Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “It is also possible to combine different compression approaches to compound their beneﬁts, see e.g., [Kuzmin et al., 2019, Section 4.3.4].”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “We refer the interested reader to [66, 95, 132] for a thorough survey of related work in pruning/sparsity. d) Knowledge distillation : Model distillation [3, 94, 148, 173, 189, 200, 261, 262] involves training a large model and then using it as a teacher to train a more compact model.”
Bayesian Bits: Unifying Quantization and Pruning (2020, methodology): “We refer the reader to [18] and [19] for overviews of hardware-friendly quantization and compression techniques, respectively.”
Differentiable Joint Pruning and Quantization for Hardware Efficiency (2020, methodology): “t pruning while optimizing for quantization jointly to achieve 4 Y. Wang, Y.D. Lu and T. Blankevoort a ﬂexible trade-oﬀ between the two. For an overview of structured pruning methods, please refer to [16]. To jointly optimize pruning and quantization, two research works have been proposed [35][28]. ”