Winning the Lottery with Continuous Sparsification

By Pedro H. P. Savarese, Hugo Silva, Michael Maire. 2019

160 citations recorded by the source corpus.

The Lottery Ticket Hypothesis conjectures that, for a typically-sized neural network, it is possible to find small sub-networks that, when trained from scratch, match the performance of the dense counterpart given a comparable training budget. The proposed algorithm to search for winning tickets, Iterative Magnitude Pruning, consistently finds sparse sub-networks which train faster and better than the overparameterized models they were extracted from, creating potential applications to problems such as transfer learning. In this paper, we propose Continuous Sparsification, a new algorithm to search for winning tickets which continuously removes parameters from a network during training, and learns the sub-network's structure with gradient-based methods instead of relying on pruning strategies. We show empirically that our method is capable of finding tickets that are sparser than the ones found by Iterative Magnitude Pruning, while achieving higher performance when trained from scratch. Moreover, our method can be efficiently parallelized, decreasing the ticket search cost measured in wall-clock time significantly given enough parallel computing resources.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, background): “Other efforts use rewinding to further study lottery tickets (Yu et al., 2020; Frankle et al., 2020; Caron et al., 2020; Savarese et al., 2020; Yin et al., 2020).”
Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning (2021, background): “In addition to image classification (Frankle & Carbin, 2019; Liu et al., 2019; Wang et al., 2020; Evci et al., 2019; Frankle et al., 2020; Savarese et al., 2020; Yin et al., 2020; You et al., 2020), LTH has been explored widely in numerous contexts, such as natural language processing (Gale et al.”
Learning to Balance Specificity and Invariance for In and Out of Domain Generalization (2020, methodology): “[44] involves learning masks over parameters under a sparsity constraint to discover small subnetworks.”
Soft Threshold Weight Reparameterization for Learnable Sparsity (2020, background): “Concurrent to our work, (Savarese et al., 2019; Liu et al., 2020; Lee, 2019; Xiao et al., 2019) have proposed learnable sparsity methods through training of the sparse masks and weights simultaneously with minimal heuristics.”
Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, background): “, 2018a) or a partial converged training time (partial dense over-parameterization) (Louizos et al., 2017; Zhu & Gupta, 2017; Gale et al., 2019; Savarese et al., 2019; Kusupati et al., 2020; You et al., 2019).”
Playing Lottery Tickets with Vision and Language (2021, background): “The properties of LTH are widely studied for image classification [44, 15, 18, 58, 68, 74, 47], and recently, the observations of LTH have also been evidenced across other fields, such as natural language processing [19, 76, 54, 6, 9], object detection [21], generative adversarial networks [10, 31, ”