A Signal Propagation Perspective for Pruning Neural Networks at Initialization

By Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, Philip H. S. Torr. 2019

174 citations recorded by the source corpus.

Network pruning is a promising avenue for compressing deep neural networks. A typical approach to pruning starts by training a model and then removing redundant parameters while minimizing the impact on what is learned. Alternatively, a recent approach shows that pruning can be done at initialization prior to training, based on a saliency criterion called connection sensitivity. However, it remains unclear exactly why pruning an untrained, randomly initialized neural network is effective. In this work, by noting connection sensitivity as a form of gradient, we formally characterize initialization conditions to ensure reliable connection sensitivity measurements, which in turn yields effective pruning results. Moreover, we analyze the signal propagation properties of the resulting pruned networks and introduce a simple, data-free method to improve their trainability. Our modifications to the existing pruning at initialization method lead to improved results on all tested network models for image classification tasks. Furthermore, we empirically study the effect of supervision for pruning and demonstrate that our signal propagation perspective, combined with unsupervised pruning, can be useful in various scenarios where pruning is applied to non-standard arbitrarily-designed architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DepGraph: Towards Any Structural Pruning (2023, methodology): “Mainstream pruning methods can be broadly categorized into two types: structural pruning [4,29,32,71,71] and unstructural pruning [8,28,44,49].”
Picking Winning Tickets Before Training by Preserving Gradient Flow (2020, background): “Concurrently to our work, Lee et al. (2019b) studied the pruning problem from a signal propagation perspective, and proposed to use an orthogonal initialization to ensure faithful signal propagation.”
What’s Hidden in a Randomly Weighted Neural Network? (2019, methodology): “This also differs from methods which prune at initialization and modify the weights of the discovered subnetwork [18, 17] or methods which modify a subset of the weights [25].”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “Different from the signal propagation perspective in [72], which focuses on initialization scheme, Wang et al. [56] propose Gradient Signal Preservation ( GraSP ) to exploit gradient norm after pruning to remove weights that have the least effect on the gradient ﬂow.”
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win (2020, methodology): “Lee et al. (2020) measures the singular values of the input gradients and proposes to use a dense orthogonal initialization to ensure dynamical isometry and improve one-shot pruning performance before training.”
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, background): “Very recently, works on pruning at initialization (PI) (Lee et al., 2019; 2020; Wang et al., 2020; Tanaka et al., 2020; de Jorge et al., 2021) have emerged to obtain trainable sparse neural networks before the main training process based on some salience criteria.”
Layer-adaptive Sparsity for the Magnitude-based Pruning (2020, background): “…be connected to the observation of Zhou et al. (2019) that the initial weights and ﬁnal weights of a model are highly correlated; global MP may help preserving connections with larger initial magnitudes, which play important roles in terms of signal propagation at initialization (Lee et al., 2020).”
Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, methodology): “Pruning Theory: Apart from utilizing optimization tools for pruning in Section II-D, some works view pruning from the perspective of synaptic ﬂow [180], signal propagation [181], and graph theory [182].”