Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask

By Hattie Zhou, Janice Lan, Rosanne Liu, Jason Yosinski. 2019

426 citations recorded by the source corpus.

The recent "Lottery Ticket Hypothesis" paper by Frankle & Carbin showed that a simple approach to creating sparse networks (keeping the large weights) results in models that are trainable from scratch, but only when starting from the same initial weights. The performance of these networks often exceeds the performance of the non-sparse base model, but for reasons that were not well understood. In this paper we study the three critical components of the Lottery Ticket (LT) algorithm, showing that each may be varied significantly without impacting the overall results. Ablating these factors leads to new insights for why LT networks perform as well as they do. We show why setting weights to zero is important, how signs are all you need to make the reinitialized network train, and why masking behaves like training. Finally, we discover the existence of Supermasks, masks that can be applied to an untrained, randomly initialized network to produce a model with performance far better than chance (86% on MNIST, 41% on CIFAR-10).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Weight Agnostic Neural Networks (2019, positive): “Similar to our work, a concurrent work [118] found pruned networks that can achieve image classiﬁcation accuracies that are much better than chance even with randomly initialized weights.”
Supermasks in Superposition (2020, methodology): “SupSup, diagrammed in Figure 1, is driven by two core ideas: a) the expressive power of untrained, randomly weighted subnetworks [57, 39], and b) inference of task-identity as a gradient-based optimization problem.”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, background): “” This work is among several recent papers to propose that merely sparsifying at initialization can produce high performance neural networks (Mallya et al., 2018; Zhou et al., 2019; Ramanujan et al., 2020; Evci et al., 2020).”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, background): “Followup work concentrated on ﬁnding these conﬁgurations faster (Frankle et al., 2019; Zhou et al., 2019).”
When BERT Plays the Lottery, All Tickets Are Winning (2020, background): “While NLP focused on building larger Trans-a formers, the computer vision community was exploring the lottery ticket hypothesis (Frankle and Carbin, 2019; Lee et al., 2018; Zhou et al., 2019).”
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win (2020, background): “These results explain non-random initial loss values for LT initializations (Zhou et al. 2019) and their inability to find good solutions if repelled by pruning solutions (Maene, Li, and Moens 2021).”
What’s Hidden in a Randomly Weighted Neural Network? (2019, methodology): “[33] In Figure 6 we compare the performance of edge-popup with Zhou et al .”
On the Soft-Subnetwork for Few-shot Class Incremental Learning (2022, methodology): “In our work, by selecting sub-networks (Frankle and Carbin, 2019; Zhou et al., 2019; Wortsman et al., 2019; Ramanujan et al., 2020; Kang et al., 2022; Chijiwa et al., 2022) and optimizing the sub-network parameters in a sub-space, we propose a new method to preserve learned knowledge from a base ses”
Effective Sparsification of Neural Networks with Global Sparsity Constraint (2021, positive): “Experimental results on network pruning and supermask [40] ﬁnding demonstrate that our method is much more effective than the state-of-the-art methods on both small scale datasets and large scale datasets and can outperform them with a signiﬁcant margin when the pruning rate is high.”
Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP (2019, background): “The lottery ticket hypothesis [8, 9, 32] is an interesting and surprising phenomenon in which small, sparse sub-networks can be found in over-parameterized deep neural networks (DNNs) which, when trained in isolation, can achieve similar or even greater performance than the original, highly overpara”
Provable and Efficient Continual Representation Learning (2022, background): “Zhou et al. (2019); Malach et al. (2020); Pensia et al. (2020) demonstrate that there are subsets with good test performance even without any training and provide theoretical guarantees.”
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020, background): “Investigating the lottery ticket hypothesis (Fran-kle and Carbin, 2018) of network pruning (Han et al., 2015a; He et al., 2018; Liu et al., 2019d; Lee et al., 2019; Lin et al., 2020), Zhou et al. (2019) ﬁnd that applying binary masks to a neural network is a form of training the network.”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, neutral): “Catalyzed by [81] and inspired by Weight Agnostic Neural Networks (WANNs) [82], Ramanujan et al.”
How fine can fine-tuning be? Learning efficient language models (2020, methodology): “We treat gradient backpropagation through µ as a straight-through estimator, similar to the techniques used in Mallya et al. (2018); Zhou et al. (2019).”
Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks (2020, background): “The recent work of Zhou et al. (2019) demonstrated how it is possible to achieve non trivial performance by training binary masks on a neural network with frozen weights that were randomly chosen.”
Drawing early-bird tickets: Towards more efficient training of deep networks (2019, neutral): “(Zhou et al., 2019) discovers the existence of supermasks that can be applied to an un-trained, randomly-initialized network.”
Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs (2020, background): “However, other papers have shown that it is possible to achieve high accuracy by learning a binarized mask in which individual weights at initialization are either retained or pruned (Zhou et al., 2019; Ramanujan et al., 2019).”