Sparse Networks from Scratch: Faster Training without Losing Performance

By Tim Dettmers, Luke S. Zettlemoyer. 2019

378 citations recorded by the source corpus.

We demonstrate the possibility of what we call sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels. We accomplish this by developing sparse momentum, an algorithm which uses exponentially smoothed gradients (momentum) to identify layers and weights which reduce the error efficiently. Sparse momentum redistributes pruned weights across layers according to the mean momentum magnitude of each layer. Within a layer, sparse momentum grows weights according to the momentum magnitude of zero-valued weights. We demonstrate state-of-the-art sparse performance on MNIST, CIFAR-10, and ImageNet, decreasing the mean error by a relative 8%, 15%, and 6% compared to other sparse algorithms. Furthermore, we show that sparse momentum reliably reproduces dense performance levels while providing up to 5.61x faster training. In our analysis, ablations show that the benefits of momentum redistribution and growth increase with the depth and size of the network. Additionally, we find that sparse momentum is insensitive to the choice of its hyperparameters suggesting that sparse momentum is robust and easy to use.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

A Review of Sparse Expert Models in Deep Learning (2022, background): “These are a special case of sparse neural networks (Gale et al., 2019; Dettmers and Zettlemoyer, 2019; Evci et al., 2020) which are similar in that they only use a subset of parameters, but differ because they have potentially irregular sparsity patterns.”
Green AI (2019, background): “18 Figure 4a shows the number of parameters and FPO of several leading object recognition models, as well as their 15Some very recent work also targeted efficient training [8].”
BASE Layers: Simplifying Training of Large, Sparse Models (2021, background): “Sparse training is a line of work where traditional architectures are trained with sparse instead of dense layers and the number of parameters allowed during training is restricted to a percentage of the dense layers (Dettmers & Zettlemoyer, 2019; Evci et al., 2020; Mostafa & Wang, 2019).”
Dynamic Model Pruning with Feedback (2020, methodology): “Following the common experimental setting in related work on network pruning (Liu et al., 2019; Gale et al., 2019; Dettmers & Zettlemoyer, 2019; Mostafa & Wang, 2019), our main experiments focus on ResNet (He et al., 2016a) and WideResNet (Zagoruyko & Komodakis, 2016).”
Soft Threshold Weight Reparameterization for Learnable Sparsity (2020, methodology): “Sparse Networks From Scratch (SNFS) (Dettmers & Zettlemoyer, 2019) utilizes momentum of the weights to re-allocate weights across layers and the Rigged Lottery (RigL) (Evci et al., 2019) uses the magnitude to drop and the periodic dense gradients to regrow weights.”
Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware (2019, methodology): “, sparse evolutionary training (SET) [52], DEEP-R [4], dynamic sparse reparameterization (DSR) [55], sparse momentum [20], ST-RNNs [43], and rigged lottery (RigL) [24].”
WoodFisher: Efficient Second-Order Approximation for Neural Network Compression (2020, methodology): “[23, 24, 27, 45], which prune during regular training and can additionally allow the re-introduction of weights during training; (4) variational or regularization-based methods, e.”
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps (2020, methodology): “…Bi-LSTM: ﬁxed linear transformations (such as the ﬁxed FFT, or no transform at all [i.e. the identity]), other train-able structured layers (low-rank, circulant, and sparse [using the sparse training algorithm of Dettmers & Zettlemoyer (2019)]), and a trainable unstructured (dense) linear layer.”
Picking Winning Tickets Before Training by Preserving Gradient Flow (2020, background): “Srinivas & Babu (2016) Dettmers & Zettlemoyer (2019), follow a prune-redistribute-regrowth cycle for pruning.”
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, methodology): “…works further introduced weight redistribution (Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019; Liu et al., 2021), gradient-based weight growth (Dettmers & Zettlemoyer, 2019; Evci et al., 2020a), and extra weights update in the backward pass (Raihan & Aamodt, 2020; Jayakumar et al., 2020) to…”
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win (2020, background): “Dettmers and Zettlemoyer (2019) proposed using momentum values, whereas Evci et al. (2020) used gradient estimates directly to guide the selection of new connections, reporting results that are on par with pruning algorithms, and has been applied to vision transformers (Chen et al. 2021), language…”
Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch (2021, methodology): “Dettmers & Zettlemoyer (2019) uses the momentum of each parameter as the criterion for growing weights and receives an improvement in test accuracy.”
Comparing Rewinding and Fine-tuning in Neural Network Pruning (2020, background): “We hope that further work on the optimization of sparse neural networks can shed light on why learning rate rewinding achieves higher accuracy than standard ﬁne-tuning and can help derive other techniques for the training of sparse networks (Smith, 2017; Dettmers & Zettlemoyer, 2019).”