DSD: Regularizing Deep Neural Networks with Dense-Sparse-Dense Training Flow

By Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Shijian Tang, Erich Elsen, Bryan Catanzaro, John Tran, William (Bill) J. Dally. 2016

83 citations recorded by the source corpus.

Modern deep neural networks have a large number of parameters, making them very powerful machine learning systems. A critical issue for training such large networks on large-scale data-sets is to prevent overfitting while at the same time providing enough model capacity. We propose DSD, a dense-sparse-dense training flow, for regularizing deep neural networks. In the first D step, we train a dense network to learn which connections are important. In the S step, we regularize the network by pruning the unimportant connections and retrain the network given the sparsity constraint. In the final D step, we increase the model capacity by freeing the sparsity constraint, re-initializing the pruned parameters, and retraining the whole dense network. Experiments show that DSD training can improve the performance of a wide range of CNN, RNN and LSTMs on the tasks of image classification, caption generation and speech recognition. On the Imagenet dataset, DSD improved the absolute accuracy of AlexNet, GoogleNet, VGG-16, ResNet50, ResNet-152 and SqueezeNet by a geo-mean of 2.1 points (Top-1) and 1.4 points (Top-5). On the WSJ’92 and WSJ’93 dataset, DSD improved DeepSpeech2 WER by 0.53 and 1.08 points. On the Flickr-8K dataset, DSD improved the NeuralTalk BLEU score by 2.0 points. DSD training flow produces the same model architecture and doesn’t incur any inference overhead.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Dynamic Model Pruning with Feedback (2020, methodology): “For instance, one-shot pruning strategies find a suitable sparsity mask by inspecting the weights of a pretrained network (Mozer & Smolensky, 1989; LeCun et al., 1990; Han et al., 2017).”
MaskConnect: Connectivity Learning by Gradient Descent (2018, positive): “Several authors have proposed learning connectivity by pruning unimportant weights from the network [26,27,28,29,30].”
Connectivity Learning in Multi-Branch Networks (2017, methodology): “Our approach is also related to methods that learn connectivity by pruning unimportant weights from the network (LeCun et al., 1989; Han et al., 2015a;b; Guo et al., 2016; Han et al., 2016).”
Importance Estimation for Neural Network Pruning (2019, background): “[7] re-initialize weights after pruning and finetune them to achieve even better accuracy than the initial model.”
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size (2016, background): “We divide this exploration into three main topics: microarchitectural exploration (per-module layer dimensions and conﬁgurations), macroarchitectural exploration (high-level end-to-end organization of modules and other layers), and exploration of model compression conﬁgurations.”
Hu-Fu: Hardware and Software Collaborative Attack Framework Against Neural Networks (2018, methodology): “Our training process, shown in Algorithm 1, is inspired by [30], which proposed training a dense and sparse CNN alternatively to improve its accuracy.”
NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm (2017, background): “This can also assist with avoiding local minimum [47].”
MobileNetV2: Inverted Residuals and Linear Bottlenecks (2018, methodology): “methods of network pruning [12, 13, 14, 15, 16, 17] and connectivity learning [18, 19].”
NeuralScale: Efficient Scaling of Neurons for Resource-Constrained Deep Neural Networks (2020, methodology): “There are two ways to prune a network: structured pruning [32, 20, 39, 28, 34, 52] and unstructured pruning [13, 14, 15, 9].”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “Han et al. (2017) and Jin et al. (2016) restore pruned connections to increase network capacity after small weights have been pruned and surviving weights fine-tuned.”
Power Law in Sparsified Deep Neural Networks (2018, methodology): “In this section, we use the (dense) VGG-16 model8 (with 13 convolutional layers and 3 fully connected layers) obtained by the dense-sparse-dense (DSD) procedure of [22].”
Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation (2018, methodology): “Recently there has been lots of progress in algorithmic architecture exploration included hyper-parameter optimization [8, 9, 10] as well as various methods of network pruning [11, 12, 13, 14, 15, 16] and connectivity learning [17, 18].”
ICNet for Real-Time Semantic Segmentation on High-Resolution Images (2017, background): “The solutions [11, 8, 9, 13] can update a complicated network to a lighter one under user-controlled accuracy reduction.”
Accelerating Sparse Deep Neural Networks (2021, methodology): “Iterative methods prune weights gradually over a number of steps, in each step eliminating either a fixed number of weights [6, 7] or choosing a fraction of weights based on an analytical function [9].”
HydraNets: Specialized Dynamic Architectures for Efficient Inference (2018, methodology): “The hyper parameters k, nb and the point to branch were chosen empirically by searching over nb = [5, 10, 20, 25], k = [3, 4, 5] and branching at block 2 or block 3.”