Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization

By Hesham Mostafa, Xin Wang. 2019

357 citations recorded by the source corpus.

Modern deep neural networks are typically highly overparameterized. Pruning techniques are able to remove a significant fraction of network parameters with little loss in accuracy. Recently, techniques based on dynamic reallocation of non-zero parameters have emerged, allowing direct training of sparse networks without having to pre-train a large dense model. Here we present a novel dynamic sparse reparameterization method that addresses the limitations of previous techniques such as high computational cost and the need for manual configuration of the number of free parameters allocated to each layer. We evaluate the performance of dynamic reallocation methods in training deep convolutional networks and show that our method outperforms previous static and dynamic reparameterization methods, yielding the best accuracy for a fixed parameter budget, on par with accuracies obtained by iteratively pruning a pre-trained dense model. We further investigated the mechanisms underlying the superior generalization performance of the resultant sparse networks. We found that neither the structure, nor the initialization of the non-zero parameters were sufficient to explain the superior performance. Rather, effective learning crucially depended on the continuous exploration of the sparse network structure space during training. Our work suggests that exploring structural degrees of freedom during training is more effective than adding extra parameters to the network.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, background): “More recently, the Lottery Ticket Hypothesis [12] has sparked interest in techniques that provide the storage and computation benefits of sparse models directly during training [35, 8].”
Initialization and Regularization of Factorized Neural Layers (2021, background): “Best of (Mostafa & Wang, 2019; Mocanu et al., 2018; Bellec et al., 2018), obtained from Wang et al. (2020).”
BASE Layers: Simplifying Training of Large, Sparse Models (2021, background): “Sparse training is a line of work where traditional architectures are trained with sparse instead of dense layers and the number of parameters allowed during training is restricted to a percentage of the dense layers (Dettmers & Zettlemoyer, 2019; Evci et al., 2020; Mostafa & Wang, 2019).”
Dynamic Model Pruning with Feedback (2020, methodology): “In dynamic pruning methods, the sparsity mask is readjusted during training according to different criteria (Mostafa & Wang, 2019; Mocanu et al., 2018).”
Rigging the Lottery: Making All Tickets Winners (2019, background): “The second sub-group includes DSR and networks with ERK sparsity distribution which require a higher number of FLOPs for inference with same parameter count.”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, methodology): “The most closely related work to ours is Dynamic Sparse Reparameterization (DSR) by Mostafa and Wang (2019), which includes the full prune-redistribute-regrowth cycle.”
Effective Sparsification of Neural Networks with Global Sparsity Constraint (2021, neutral): “Sparse-to-sparse training starts with a sparse network and maintain the sparsity during training [1, 26, 28, 5, 4].”
WoodFisher: Efficient second-order approximations for model compression (2020, methodology): “Other variants of this approach have tried to correct for non-optimally removed parameters via magnitude-pruning by using some re-introduction heuristics as in Guo et al. (2016); Mostafa and Wang (2019).”
SpaceNet: Make Free Space For Continual Learning (2020, background): “In recent years, sparse training proved its success in achieving the same performance with dense neural networks for single task standard supervised/unsupervised learning, while having much faster training speed and much lower memory requirements [30, 2, 4, 5, 14, 32].”
Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers (2020, methodology): “The state-of-the-art algorithms, DEEP-R (Bellec et al., 2017), SET (Mocanu et al., 2018), DSR (Mostafa & Wang, 2019), and Sparse momentum (Dettmers & Zettlemoyer, 2019), are selected for comparsion.”
Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware (2019, methodology): “, sparse evolutionary training (SET) [52], DEEP-R [4], dynamic sparse reparameterization (DSR) [55], sparse momentum [20], ST-RNNs [43], and rigged lottery (RigL) [24].”
Low-Memory Neural Network Training: A Technical Report (2019, methodology): “…search both the frequency at which to rewire in { 0.125, 0.25, 0.5, 1, 2, 4 } times the default frequency from (Mostafa and Wang, 2019) and fraction of parameters to rewire in { 0.25, 0.5, 1, 2, 4, 8 } times the default fraction from (Mostafa and Wang, 2019) (see Appendix A.2 for more details).”
Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch (2021, methodology): “Before the advent of N : M ﬁne-grained structured sparsity, there exist many state-of-the-art methods to generate sparsity models, including DSR (Mostafa & Wang, 2019), RigL (Evci et al., 2019a), GMP (Gale et al., 2019), and STR (Kusupati et al., 2020).”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, neutral): “For example, Su et al. [55] propose randomly pruning each layer by using a series of layerwise keep-ratios (smart-ratios) without using any data.”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, neutral): “Sparse Language Models Sparsely activated language models have been considered in a few forms (Evci et al., 2020; Mostafa and Wang, 2019; Dettmers and Zettlemoyer, 2019), but the Mixture-of-Experts (MoE) model is of particular note.”
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps (2020, methodology): “There is also a more nascent line of work that aims to train unstructured sparse neural networks directly (Mocanu et al., 2018; Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019; Evci et al., 2019).”
WoodFisher: Efficient Second-Order Approximation for Neural Network Compression (2020, neutral): “Relative Drop Sparsity Method Dense ( D ) Pruned ( P ) 100 DSR [49] 74.90 71.60 -4.41 80.00 Incremental [19] 75.95 74.25 -2.24 73.50 DPF [24] 75.95 75.13 -1.08 79.90 GMP + LS [18] 76.69 75.58 -1.44 79.90 VD [44, 46] 76.69 75.28 -1.83 80.00 RIGL + ERK [45] 76.80 75.10 -2.21 80.00 SNFS + LS [23]…”
Soft Threshold Weight Reparameterization for Learnable Sparsity (2020, methodology): “Dynamic Sparse Reparameterization (DSR) (Mostafa & Wang, 2019) heuristically obtains a global magnitude threshold along with the re-allocation of the weights based on the non-zero weights present at every step.”