SNIP: Single-shot Network Pruning based on Connection Sensitivity

By Namhoon Lee, Thalaiyasingam Ajanthan, Philip H. S. Torr. 2018

1,499 citations recorded by the source corpus.

Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Initialization and Regularization of Factorized Neural Layers (2021, methodology): “Best of (Lee et al., 2018; Wang et al., 2020; Su et al., 2020), obtained from Wang et al. (2020) and Su et al. (2020).”
What is the State of Neural Network Pruning? (2020, methodology): “Others consider scores globally, comparing scores to one another irrespective of the part of the network in which the parameter resides (Lee et al., 2019b; Frankle & Carbin, 2019).”
Weight Agnostic Neural Networks (2019, background): “mage classiﬁcation tasks compared to the full network. By retaining the original weight initialization values, these sparse networks can even be trained from scratch to achieve a higher test accuracy [23, 65] than the original network. Similar to our work, a concurrent work [127] found pruned networ”
Dynamic Model Pruning with Feedback (2020, background): “tly—spurred by the lottery ticket hypothesis (LT) (Frankle & Carbin, 2019)—methods which try to ﬁnd a sparse mask that can be trained from scratch have attracted increased interest. For instance, Lee et al. (2019) propose SNIP to ﬁnd a pruning mask by inspecting connection sensitivities and iden”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “In one-shot NAS, redundant architecture parameters are pruned based on the magnitude of weights similar to that used in Network Compression.”
Rigging the Lottery: Making All Tickets Winners (2019, methodology): “A Tensorﬂow (Abadi et al., 2015) implementation of our method along with three other baselines (SET, SNFS, SNIP) and check-points of our models can be found at github.com/google-research/rigl.”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, methodology): “For comparison, we follow three different experimental settings, one from Lee et al. (2019) and two settings follow Mostafa and Wang (2019): For MNIST (LeCun, 1998), we use a batch size of 100, decay the learning rate by a factor of 0.1 every 25000 mini-batches.”
When BERT Plays the Lottery, All Tickets Are Winning (2020, background): “formers, the computer vision community was exploring the Lottery Ticket Hypothesis (LTH: Frankle and Carbin, 2019; Lee et al., 2018; Zhou et al., 2019).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Pruning methods can be broadly categorized into unstructured pruning [49, 85, 137, 141, 186, 249], and structured pruning [90, 105, 153, 162, 266, 267, 271].”
The Lottery Ticket Hypothesis for Pre-trained BERT Networks (2020, background): “In parallel to this race for ever-larger models, an emerging subﬁeld has explored the prospect of training smaller subnetworks in place of the full models without sacriﬁcing performance [10–15].”
Picking Winning Tickets Before Training by Preserving Gradient Flow (2020, methodology): “Lee et al. (2018) proposed the ﬁrst algorithm for pruning at initialization time: Single-shot Network Pruning (SNIP), which uses a connection sensitivity criterion to prune weights with both small magnitude and small gradients.”
A Signal Propagation Perspective for Pruning Neural Networks at Initialization (2019, background): “Another important point to consider is that, in SNIP [7], a global threshold is chosen based on CS to obtain the pruned network (refer Section 2), which requires the CS scores to be of the same scale for each layer.”
Discrimination-Aware Network Pruning for Deep Model Compression (2020, background): “In order to measure the importance of the network module, different metrics [23], [26], [37], [38], [51], [57], [81], [82] were proposed.”
A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “Pruning Method Pruning Type Pruning Metric Training Procedure SNIP (Lee et al., 2018) Unstructured Loss Sensitivity Pruning at Initialization BERT-LTH (Chen et al.”
ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks (2018, methodology): “In this context, existing approaches can be roughly grouped into two categories: (i) parameter pruning and sharing [7, 9, 14, 16, 31, 32, 38, 55], which aims to remove the least informative parameters, yielding an arbitrary compact network with information loss; and (ii) low-rank matrix factorizatio”
Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch (2021, methodology): “Many different pruning criterions and pruning meth-ods are proposed for unstructured sparsity, e . g ., magnitude-based pruning (Han et al., 2015; Frankle & Carbin, 2018), Hessian based heuristics (LeCun et al., 1990), and pruning with connection sensitivity (Lee et al., 2018).”
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win (2020, background): “While the majority of pruning algorithms focus on pruning after training, a subset focuses on pruning NNs before training (Lee, Ajanthan, and Torr 2019; Wang, Zhang, and Grosse 2020; Tanaka et al. 2020).”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “Because of this simpler magnitude-based algorithms have been developed [167, 132].”