DSD: Dense-Sparse-Dense Training for Deep Neural Networks

By Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Enhao Gong, Shijian Tang, Erich Elsen, Péter Vajda, Manohar Paluri, John Tran and 2 more. 2016

219 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning (2019, neutral): “noting the weights retained from the ﬁrst and second incremental step respectively. performance. Network pruning. Considerable research has explored this area to reduce network redundancy. Han et al. [10,11] propose to compress network through quantization and Huffman coding. Yu et al. [36] compress”
CLIP-Q: Deep Network Compression Learning by In-parallel Pruning-Quantization (2018, methodology): “Besides compression, network pruning has also been used to regularize the training of more accurate full-sized networks [17], reduce over-ﬁtting in transfer learning [31], and produce energy-efﬁcient networks for battery-powered devices [55].”
SPACE: Structured Compression and Sharing of Representational Space for Continual Learning (2020, methodology): “Leveraging the fact that DNNs are overparameterized [7], PackNet employs weight magnitude based iterative pruning [12, 11] to free up unimportant parame-”
Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks With Octave Convolution (2019, methodology): “Since OctConv mainly focuses on processing feature maps at multiple spatial frequencies and reducing their spatial redundancy, it is orthogonal and complementary to existing methods that focus on building better CNN topology [22, 41, 35, 33, 29], reducing channel-wise redundancy in convolutional fea”
Distilling Knowledge From a Deep Pose Regressor Network (2019, methodology): “This approach promises significant parameter reduction without greatly affecting accuracy, although it typically requires more training stages [13].”
Visual Attention Methods in Deep Learning: An In-Depth Survey (2022, background): “features [60], [176], [177] and consequently, the discrimination power for feature representation will not be diverse.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, background): “Dense-Sparse-Dense (DSD) training [87] helps various network improve classification accuracy by 1.”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, background): “For example, sparsity can be used to regularize dense models to improve accuracy (Han et al., 2016), or to alternate between sparse and dense training to ease deployment (Peste et al.”