Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers

By Junjie Liu, Zhe Xu, Runbin Shi, Ray Chak-Chung Cheung, Hayden Kwok-Hay So. 2020

140 citations recorded by the source corpus.

We present a novel network pruning algorithm called Dynamic Sparse Training that can jointly find the optimal network parameters and sparse network structure in a unified optimization process with trainable pruning thresholds. These thresholds can have fine-grained layer-wise adjustments dynamically via backpropagation. We demonstrate that our dynamic sparse training algorithm can easily train very sparse neural network models with little performance loss using the same training epochs as dense models. Dynamic Sparse Training achieves prior art performance compared with other sparse training algorithms on various network architectures. Additionally, we have several surprising observations that provide strong evidence to the effectiveness and efficiency of our algorithm. These observations reveal the underlying problems of traditional three-stage pruning algorithms and present the potential guidance provided by our algorithm to the design of more compact network architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Prateek Jain
2. Sham M. Kakade
3. Ali Farhadi
4. Mitchell Wortsman
5. C. Dovrolis
6. Mustafa Burak Gurbuz
7. Mykola Pechenizkiy
8. Aditya Kusupati

In Computer Vision and Pattern Recognition

1. Mitchell Wortsman
2. Ali Farhadi
3. Luc Van Gool
4. Dacheng Tao
5. Prateek Jain
6. Aditya Kusupati
7. Trevor Darrell
8. Kristen Grauman

In Signal Processing

1. Ali Farhadi
2. Sham M. Kakade
3. Mykola Pechenizkiy
4. Decebal Constantin Mocanu
5. Prateek Jain
6. Raghav Somani
7. Mitchell Wortsman
8. Aditya Kusupati

Citations in the corpus

• NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (2022, background): “Training sparse neural networks has been extensively explored (Bellec et al., 2018; Evci et al., 2020; Mocanu et al., 2018; Liu et al., 2021b; 2020; 2021a).”
• A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “DST (2020) [68] min w,t l(y, f(x;w)) + λ ∑L i=1 ∑c(i) out j=1 exp(−tj) N S”
• SpaceNet: Make Free Space For Continual Learning (2020, background): “In recent years, sparse training proved its success in achieving the same performance with dense neural networks for single task standard supervised/unsupervised learning, while having much faster training speed and much lower memory requirements [30, 2, 4, 5, 14, 32].”
• Loss of plasticity in deep continual learning (2024, unknown): “Various papers [61, 62, 63] have proposed different measures of connection utility for the network pruning problem.”
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, background): “Liu et al. (2020c) found that DST gradually optimizes the initial sparse topology towards a completely different one.”
• Soft Threshold Weight Reparameterization for Learnable Sparsity (2020, background): “Concurrent to our work, (Savarese et al., 2019; Liu et al., 2020; Lee, 2019; Xiao et al., 2019) have proposed learnable sparsity methods through training of the sparse masks and weights simultaneously with minimal heuristics.”