NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm

By Xiaoliang Dai, Hongxu Yin, Niraj Kumar Jha. 2017

258 citations recorded by the source corpus.

Deep neural networks (DNNs) have begun to have a pervasive impact on various applications of machine learning. However, the problem of finding an optimal DNN architecture for large applications is challenging. Common approaches go for deeper and larger DNN architectures but may incur substantial redundancy. To address these problems, we introduce a network growth algorithm that complements network pruning to learn both weights and compact DNN architectures during training. We propose a DNN synthesis tool (NeST) that combines both methods to automate the generation of compact and accurate DNNs. NeST starts with a randomly initialized sparse network called the seed architecture. It iteratively tunes the architecture with gradient-based growth and magnitude-based pruning of neurons and connections. Our experimental results show that NeST yields accurate, yet very compact DNNs, with a wide range of seed architecture selection. For the LeNet-300-100 (LeNet-5) architecture, we reduce network parameters by $70.2\times$70.2× ($74.3\times$74.3×) and floating-point operations (FLOPs) by $79.4\times$79.4× ($43.7\times$43.7×). For the AlexNet, VGG-16, and ResNet-50 architectures, we reduce network parameters (FLOPs) by $15.7\times$15.7× ($4.6\times$4.6×), $33.2\times$33.2× ($8.9\times$8.9×), and $4.1\times$4.1× ($2.1\times$2.1×) respectively. NeST's grow-and-prune paradigm delivers significant additional parameter and FLOPs reduction relative to pruning-only methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Neural Architecture Search & AutoML

Citations in the corpus

AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “Structure pruning and growing were proposed for different goals, such as efficient inference [7, 8, 11, 13, 14, 18, 19, 24, 25, 27, 38–40], lifelong learning [41] and model adaptation [10, 29].”
Dynamic Model Pruning with Feedback (2020, methodology): “Sparse evolutionary training (SET) (Mocanu et al., 2018) simplifies prune–regrowth cycles by using heuristics for random growth at the end of each training epoch and NeST (Dai et al., 2019) by inspecting gradient magnitudes.”
FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, methodology): “Conv 3 (16, 24, 2) 1 2 hswish MBConv [3, 5] 1 (16, 24, 2) (1, 4) 1 N hswish MBConv [3, 5] (4, 7) / (2, 5) (20, 32, 4) (4, 7) 2 N hswish MBConv [3, 5] (4, 7) / (2, 5) (24, 48, 4) (4, 7) 2 Y hswish MBConv [3, 5] (4, 7)(1) / (2, 5)(2) (56, 84, 4) (4, 8) 2 N hswish MBConv [3, 5] (4, 7)(1) / (2, 5)(2) (9”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, neutral): “…et al., 2018), continuous pruning and re-training (Guo et al., 2016), joint loss/pruning-cost optimization (Carreira-Perpinán and Idelbayev, 2018), layer-by-layer pruning (Dong et al., 2017), fast-switching growth-pruning cycles (Dai et al., 2017), and soft weight-sharing (Ullrich et al., 2017).”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, methodology): “Note that an intermediate case is NeST (Dai et al., 2017; 2018), which starts training with a small network, grows it to a large size, and finally prunes it down again.”
ChamNet: Towards Efficient Network Design Through Platform-Aware Model Adaptation (2018, methodology): “Network pruning [11, 29, 6, 34, 32, 7] has been a popular approach for removing redundancy in NNs.”
Designing Adaptive Neural Networks for Energy-Constrained Image Classification (2018, neutral): “First, prior art has investigated pruning-based methods to reduce the network connections [6, 11, 33].”
A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers (2018, methodology): “There is recent work [5] that employs an evolutionary algorithm for weight pruning, which incorporates randomness in both pruning and growing of weights, following certain probabilistic rules.”
Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM (2019, methodology): “A grow-and-prune paradigm typically reduces the number of parameters in CNNs [13] and LSTMs [18] by another 2 × .”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Given a small seed network, NeST [217] combines the gradient-aware growing and magnitude-aware pruning, which can be used on both weights and neurons.”
CGaP: Continuous Growth and Pruning for Efficient Deep Learning (2019, background): “[Dai et al., 2017] picked a set of convolutional filters from a bundle of randomly generated ones to enlarge the convolutional layers.”
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, methodology): “By relaxing the constraint of the fixed parameter count, Dai et al. (2019; 2018b) proposed a Grow-and-Prune strategy based on gradient-based growth and magnitude-based pruning to yield an accurate, yet very compact sparse network.”
Fully Dynamic Inference With Deep Neural Networks (2020, unknown): “r of parameters of VGG-16 can be reduced by more than 13 without any accuracy loss [13]. A recent work combines pruning with network growth and improves the compression ratio of VGG-16 by another 2.5 [20]. Furthermore, structured sparsity and pruning can signiﬁcantly reduce run-time latency [21]. Lo”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “Magnitude-based criteria can be applied to either unstructured ([48], [49], [153]) or structured pruning ([63], [94], [132]).”
Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware (2019, methodology): “The Nest algorithm (Dai et al., 2019) gets rid of a fully-connected network at the beginning by a grow-and-prune paradigm, that is, expanding a small randomly initialized sparse network to a large one and then shrink it down.”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, background): “For example, in [95], in each iteration, a small percentage of the least important weights based on a predeﬁned threshold are removed from every layer, and the models are retrained to evaluate the performance.”