Pruning neural networks without any data by iteratively conserving synaptic flow

By Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, Surya Ganguli. Neural Information Processing Systems, 2020

844 citations recorded by the source corpus.

Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy both during training and at test time. Recent works have identified, through an expensive sequence of training and pruning cycles, the existence of winning lottery tickets or sparse trainable subnetworks at initialization. This raises a foundational question: can we identify highly sparse trainable subnetworks at initialization, without ever training, or indeed without ever looking at the data? We provide an affirmative answer to this question through theory driven algorithm design. We first mathematically formulate and experimentally verify a conservation law that explains why existing gradient-based pruning algorithms at initialization suffer from layer-collapse, the premature pruning of an entire layer rendering a network untrainable. This theory also elucidates how layer-collapse can be entirely avoided, motivating a novel pruning algorithm Iterative Synaptic Flow Pruning (SynFlow). This algorithm can be interpreted as preserving the total flow of synaptic strengths through the network at initialization subject to a sparsity constraint. Notably, this algorithm makes no reference to the training data and consistently competes with or outperforms existing state-of-the-art pruning algorithms at initialization over a range of models (VGG and ResNet), datasets (CIFAR-10/100 and Tiny ImageNet), and sparsity constraints (up to 99.99 percent). Thus our data-agnostic pruning algorithm challenges the existing paradigm that, at initialization, data must be used to quantify which synapses are important.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/3b0fb765716ef6861a84abffcbe40643857c613b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook (2021, background): “Various forms of network compression developed recently [130] save both energy, time, and memory resources that are especially precious for computein-memory architectures.”
Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot (2020, methodology): “Recently [40] also observe this phenomenon and propose a pruning method without data and be able to avoid the whole-layer collapse.”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, methodology): “As a model compression technique, quantization is promising compared to other methods, such as network pruning (Tanaka et al., 2020; Li et al., 2021; Ma et al., 2020; 2021a; Yuan et al., 2021) and slimming (Liu et al., 2017; 2018), as it achieves a large compression ratio (Krishnamoorthi, 2018;…”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “1 Pruning Topics Theory [110], [179], [180], [181], [182], [183], [184], [185], [186], [187], [188], Mechanism [189], [190], [191], Rate [192], [193], Domain [64], [194]”
Zero-Cost Proxies for Lightweight NAS (2021, methodology): “This is inspired by recent pruning-at-initialization work by Lee et al. (2019), Wang et al. (2020) and Tanaka et al. (2020) wherein a per-parameter saliency metric is computed before training to inform parameter pruning.”
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models (2021, background): “In an attempt, follow-up works [Wang et al., 2020, Tanaka et al., 2020] show that one can find tickets without training labels.”
Model Pruning Enables Efficient Federated Learning on Edge Devices (2019, methodology): “There are also one-shot pruning approaches including SynFlow [17] that prunes the model at model initialization (before training), and SNIP [18] that prunes the model using first training round’s gradient information.”
Learning Neural Network Subspaces (2021, methodology): “For Tiny-ImageNet we use the ResNet 18 , 50 models which were used by Tanaka et al. (2020) in their TinyImageNet experiments.”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, background): “Furthermore, follow-up works (Liu & Zenke, 2020; Wang et al., 2020a; Tanaka et al., 2020) show that we can find tickets without any training labels.”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, neutral): “5, and run-time pruning.”
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win (2020, background): “While the majority of pruning algorithms focus on pruning after training, a subset focuses on pruning NNs before training (Lee, Ajanthan, and Torr 2019; Wang, Zhang, and Grosse 2020; Tanaka et al. 2020).”
pTNAS: Progressive Neural Architecture Search for Tabular Data (2024, unknown): “Consequently, the trainability of an architecture can be characterized by the global aggregation of these scores (Tanaka et al., 2020; Mellor et al., 2021): Expressivity characterizes the hypothesis space that an architecture can realize, specifically, the complexity of the input-to-output mappings…”