Runtime Neural Pruning

By Ji Lin, Yongming Rao, Jiwen Lu, Jie Zhou. 2017

528 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

The State of Sparsity in Deep Neural Networks (2019, background): “Reinforcement learning has also been applied to automatically prune weights and convolutional ﬁlters (Lin et al., 2017; He et al., 2018), and a number of techniques have been proposed that draw inspiration from biological phenomena, and derive from evolutionary algorithms and neuromorphic computing…”
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, methodology): “Sparse models have seen success in compressing models for inference (pruning) by sparsifying the weight matrices [25, 40, 41, 57, 79].”
Dynamic Neural Networks: A Survey (2021, methodology): “Commonly seen reward signals are usually constructed to minimize a penalty item of the computational cost [15], [47].”
Once for All: Train One Network and Specialize it for Efficient Deployment (2019, background): “For example, [17, 26, 28] propose to learn an additional controller or gating modules to adaptively drop blocks in a given neural network; [13] introduces early-exit branches in the computation graph, allowing to exit in the middle based on the current prediction confidence; [15] proposes to adaptiv”
AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, methodology): “RNP [33] groups all convolutional channels into 4 sets and trains a RL agent to decide on the 4 sets according to input image.”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, background): “Another way is to prune, quantize and compress an existing model for efficient deployment [16, 15, 29, 59, 18, 47].”
HAQ: Hardware-Aware Automated Quantization With Mixed Precision (2018, methodology): “For instance, the coarse-grained channel pruning methods [12, 19, 21] prune away the entire channel of convolution kernels to achieve speedup.”
Compressing Neural Networks using the Variational Information Bottleneck (2018, methodology): “Finally, (Lin et al., 2017) introduces a radically diﬀerent approach based on reinforcement learning while (Li et al., 2016) speciﬁcally prunes convolutional neural network ﬁlters that have small norms.”
Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification (2020, background): “Some other works propose to dynamically skip unnecessary layers [52, 55, 60] or channels [33].”
Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism (2022, unknown): “It has been generally used to choose over different channels in dynamic pruning methods [4,8,10,13,19,21,29] and dynamic slimmable network [25].”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, methodology): “Like RNP [123], DRLP uses reinforcement learning (RL).”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, methodology): “[66] prunes the network at runtime, hence retaining the flexibility of the full model.”
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models (2021, methodology): “Subsequent methods [Li et al., 2016, Lin et al., 2017, Dong et al., 2017, Sanh et al., 2020, Lagunas et al., 2021, Zhu and Gupta, 2017] improve on the quality of the pruned models.”
Self-Adaptive Network Pruning (2019, methodology): “Instead of using the entire feed forward graph of the network, dynamic path networks Lin et al. (2017)Liu & Deng (2018)Hua et al. (2018)Gao et al. (2019) selectively execute a subset of modules at inference time based on input samples.”
Adversarial Robustness vs. Model Compression, or Both? (2019, background): “There are existing irregular pruning work [17, 15, 47, 48] and regular pruning work [20, 42, 49, 25, 29].”
Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds (2018, background): “, by embedding sparsity as a constraint [24, 3, 36].”