Picking Winning Tickets Before Training by Preserving Gradient Flow

By Chaoqi Wang, Chaoqi Wang, Guodong Zhang, Roger Baker Grosse. 2020

770 citations recorded by the source corpus.

Overparameterization has been shown to benefit both the optimization and generalization of neural networks, but large networks are resource hungry at both training and test time. Network pruning can reduce test-time resource requirements, but is typically applied to trained networks and therefore cannot avoid the expensive training process. We aim to prune networks at initialization, thereby saving resources at training time as well. Specifically, we argue that efficient training requires preserving the gradient flow through the network. This leads to a simple but effective pruning criterion we term Gradient Signal Preservation (GraSP). We empirically investigate the effectiveness of the proposed method with extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet, using VGGNet and ResNet architectures. Our method can prune 80% of the weights of a VGG-16 network on ImageNet at initialization, with only a 1.6% drop in top-1 accuracy. Moreover, our method achieves significantly better performance than the baseline at extreme sparsity levels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Initialization and Regularization of Factorized Neural Layers (2021, methodology): “Best of (Frankle & Carbin, 2019; Renda et al., 2020; Su et al., 2020), obtained from Wang et al. (2020) and Su et al. (2020).”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, neutral): “Recent proposals attempt to prune networks at initialization (Lee et al., 2019; Wang et al., 2020), but our results suggest that the best time to do so may be after some training.”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, methodology): “Different means of achieving unstructured LTH have been proposed: SNIP [268] by preserving the training loss; GraSP [269] by preserving the gradient flow; Neural Tangent Kernel (NTK) [270] by capturing the training dynamics; GF [271] by pruning weights that cause the least change to the gradient flo”
Pruning neural networks without any data by iteratively conserving synaptic flow (2020, methodology): “However, recent works [32, 14, 33] have identiﬁed a key failure mode, layer-collapse , for existing pruning algorithms using global-masking.”
Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (2021, methodology): “Inspired by recent works on pruning-from-scratch (Lee et al., 2018; Wang et al., 2020), we propose a pruning-by-importance NAS mechanism to quickly shrink the search possibilities and boost the 1We tried some weighted summations of the two, and find their equal-weight summation to perform the…”
Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning (2021, background): “In addition to image classification (Frankle & Carbin, 2019; Liu et al., 2019; Wang et al., 2020; Evci et al., 2019; Frankle et al., 2020; Savarese et al., 2020; Yin et al., 2020; You et al., 2020), LTH has been explored widely in numerous contexts, such as natural language processing (Gale et al.”
Effective Sparsification of Neural Networks with Global Sparsity Constraint (2021, neutral): “The performance of sparse-to-sparse training generally falls behind dense-to-sparse training under the same setting as shown in [33, 18].”
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, background): “…training), and solutions that are discovered through either a partial dense over-parameterization (pruning at initialization (Lee et al., 2019; Wang et al., 2020; de Jorge et al., 2020)) or no over-parameterization (randomly-initialized static sparse training (Mocanu et al., 2016; Evci et…”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, background): “Furthermore, follow-up works (Liu & Zenke, 2020; Wang et al., 2020a; Tanaka et al., 2020) show that we can find tickets without any training labels.”
Zero-Cost Proxies for Lightweight NAS (2021, methodology): “A new class of pruning-at-initialization algorithms, that require no training, were introduced by Lee et al. (2019) and extended by Wang et al. (2020) and Tanaka et al. (2020).”
Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot (2020, methodology): “Similar trends are also reported in Figure 3 of [41], Figure 1b of [32] and Figure 11 of [11].”
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models (2021, background): “In an attempt, follow-up works [Wang et al., 2020, Tanaka et al., 2020] show that one can find tickets without training labels.”
Playing Lottery Tickets with Vision and Language (2021, background): “The properties of LTH are widely studied for image classification [44, 15, 18, 58, 68, 74, 47], and recently, the observations of LTH have also been evidenced across other fields, such as natural language processing [19, 76, 54, 6, 9], object detection [21], generative adversarial networks [10, 31, ”
Model Pruning Enables Efficient Federated Learning on Edge Devices (2019, methodology): “To analyze the empirical risk reduction, we use a first-order approximation, which is a common practice in the literature [18], [43], [44].”
pTNAS: Progressive Neural Architecture Search for Tabular Data (2024, unknown): “Recently, Neural Architecture Search (NAS) has offered a principled approach to automate DNN architecture design and shows promise in other modalities such as vision and language (Ji et al., 2025; Kang, 2025; Wang et al., 2020b).”