Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win
By Utku Evci, Yani Ioannou, Cem Keskin, Yann Dauphin. 2020
108 citations recorded by the source corpus.
Sparse Neural Networks (NNs) can match the generalization of dense NNs using a fraction of the compute/storage for inference, and have the potential to enable efficient training. However, naively training unstructured sparse NNs from random initialization results in significantly worse generalization, with the notable exceptions of Lottery Tickets (LTs) and Dynamic Sparse Training (DST). In this work, we attempt to answer: (1) why training unstructured sparse networks from random initialization performs poorly and; (2) what makes LTs and DST the exceptions? We show that sparse NNs have poor gradient flow at initialization and propose a modified initialization for unstructured connectivity. Furthermore, we find that DST methods significantly improve gradient flow during training over traditional sparse training methods. Finally, we show that LTs do not improve gradient flow, rather their success lies in re-learning the pruning solution they are derived from — however, this comes at the cost of learning novel solutions.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mykola Pechenizkiy
- Shiwei Liu
- Lu Yin
- Decebal Constantin Mocanu
- Indrė Žliobaitė
- Faisal Kamiran
- Toon Calders
- Xintao Wu
In Computer Vision and Pattern Recognition
Citations in the corpus
- Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, neutral): “[9] when training highly sparse networks.”
- A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “Some recent works ([69], [70], [79]) explore why an effective subnetwork can be identified at initialization.”
- Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, methodology): “Evci et al. (2020b) found that sparse neural networks that are initialized by a dense initialization e.g., He et al. (2015), suffer from a poor gradient flow, whereas DST can improve the gradient flow during training significantly.”