Scalpel: Customizing DNN pruning to the underlying hardware parallelism

By Jiecao Yu, Andrew Lukefahr, David J. Palframan, Ganesh S. Dasika, Reetuparna Das, Scott A. Mahlke. 2017

105 citations recorded by the source corpus.

As the size of Deep Neural Networks (DNNs) continues to grow to increase accuracy and solve more complex problems, their energy footprint also scales. Weight pruning reduces DNN model size and the computation by removing redundant weights. However, we implemented weight pruning for several popular networks on a variety of hardware platforms and observed surprising results. For many networks, the network sparsity caused by weight pruning will actually hurt the overall performance despite large reductions in the model size and required multiply-accumulate operations. Also, encoding the sparse format of pruned networks incurs additional storage space overhead. To overcome these challenges, we propose Scalpel that customizes DNN pruning to the underlying hardware by matching the pruned network structure to the data-parallel hardware organization. Scalpel consists of two techniques: SIMD-aware weight pruning and node pruning. For low-parallelism hardware (e.g., microcontroller), SIMD-aware weight pruning maintains weights in aligned fixed-size groups to fully utilize the SIMD units. For high-parallelism hardware (e.g., GPU), node pruning removes redundant nodes, not redundant weights, thereby reducing computation without sacrificing the dense matrix format. For hardware with moderate parallelism (e.g., desktop CPU), SIMD-aware weight pruning and node pruning are synergistically applied together. Across the microcontroller, CPU and GPU, Scalpel achieves mean speedups of 3.54x, 2.61x, and 1.25x while reducing the model sizes by 88%, 82%, and 53%. In comparison, traditional weight pruning achieves mean speedups of 1.90x, 1.06x, 0.41x across the three platforms.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

Citations in the corpus

• Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (2018, neutral): “We note that pruning has been proposed in prior work for non-security reasons, specifically, to reduce the computational expense of evaluating a DNN [19,48,25,4,33].”
• Machine Learning at Facebook: Understanding Inference at the Edge (2019, unknown): “Recent work on hardware for machine learning and efficient training and inference has substantially advanced the state-of-the-art [41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66].”
• Dual Dynamic Inference: Enabling More Efficient, Adaptive, and Controllable Deep Inference (2019, unknown): “More structured pruning was exploited by enforcing group sparsity, such as the filter or channel pruning [21]–[29].”
• Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning (2020, unknown): “On computation, it is difficult to be accelerated on current GPU architectures as reported in (Han et al., 2016; Wen et al., 2016; Yu et al., 2017).”
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference (2020, unknown): “Coarser-grained channel level pruning has been explored in (Yu et al. 2017; Liu et al. 2017; He, Zhang, and Sun 2017) by enforcing group sparsity.”
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, unknown): “For example, the block-wise weight sparsity is leveraged to optimize the running performance on general-purpose processors [13], [220], [234], and a more aggressive diagonal weight matrix is utilized in the accelerator design [282].”
• AccPar: Tensor Partitioning for Heterogeneous Deep Learning Accelerators (2020, unknown): “…[138], ADM-NN [139], Gist [140] Co-Design DPS-CNN [71], Minerva [72], MoDNN [73], MeDNN [74], AdaLearner [75], Stitch-X [76], PIM-DNN [77], Scalpel [78], CirCNN [79], CoSMIC [80], SnaPEA [81], OLAcce [82], Prediction-based DNN [83], PERMDNN [84], MnnFast [141], DNN Computation Reuse [142], vDNN…”
• AutoScale: Energy Efficiency Optimization for Stochastic Edge Inference Using Reinforcement Learning (2020, unknown): “The frameworks and middleware also enable efficient inference at the edge— various NN optimizations, such as quantization [15], [30], [52], [62], [65], [111], [119], weight compression [43], [68], and graph pruning [112], [118] can be employed before the DNNs are deployed.”
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey (2017, unknown): “, SIMD) found in existing general purpose hardware, which results in more efficient processing [149]; (2) it amortizes the overhead cost required to signal the location of the non-zero weights across a group of weights, which improves compression and thus reduces storage cost.”
• MnnFast: A Fast and Scalable System Architecture for Memory-Augmented Neural Networks (2019, unknown): “The excessive approximation can threaten our safety [89], and also degrade performance and energy if subsequent tasks are inefficient on inaccurate data [73, 86, 87].”
• CompAct (2019, unknown): “Weight/filter compression techniques such as CNN/DNN pruning [13, 36, 39] are orthogonal to the goal of the CompAct, that focuses on compressing activations.”