The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
By Jonathan Frankle, Michael Carbin. 2018
4,345 citations recorded by the source corpus.
Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance. We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective. We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Quoc V. Le
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Ruslan Salakhutdinov
- Sergey Levine
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Alan L. Yuille
In Efficiency & Compression
- Dan Alistarh
- Song Han
- Jonathan Frankle
- Luke S. Zettlemoyer
- Sidak Pal Singh
- Tim Dettmers
- Michael Carbin
- Martin Jaggi
In Scaling & Learning Theory
Citations in the corpus
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “Connection to the Lottery Ticket Hypothesis Our compression findings have deep connections to recent conjectures such as the lottery ticket hypothesis (Frankle & Carbin, 2019).”
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “unevenly partitioned in 2 ways on the first dimension (partition shape [2, 2]), and the output is also partitioned in 2 ways (partition shape [3]).”
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “For model training, the lottery tickets hypothesis [30–32] suggests that there are a set of small sub-networks derived from a larger dense network that performs as well as the original dense network.”
- Pretrained Transformers for Text Ranking: BERT and Beyond (2020, neutral): “We also note the connection here with the so-called “Lottery Ticket Hypothesis” Frankle and Carbin [2019], Yu et al.”
- Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, background): “More recently, the Lottery Ticket Hypothesis [12] has sparked interest in techniques that provide the storage and computation benefits of sparse models directly during training [35, 8].”
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, background): “The remarkable success of those works have sparked interest the lottery-ticket hypothesis (Frankle and Carbin, 2019; Chen et al., 2020; Prasanna et al., 2020): the conjecture that large models are needed in pretraining only to induce (in high probability)”
- Initialization and Regularization of Factorized Neural Layers (2021, methodology): “Best of (Frankle & Carbin, 2019; Renda et al., 2020; Su et al., 2020), obtained from Wang et al. (2020) and Su et al. (2020).”
- Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, background): “We call this process “lottery ticket winning” near random initialization, which is related to the study of [34].”
- Training Neural Networks with Fixed Sparse Masks (2021, background): “This is in line with past work demonstrating the difficulty of identifying fixed sparse subnetworks to train before training begins [15].”
- A Primer in BERTology: What We Know About How BERT Works (2020, background): “(2020) explore BERT from the perspective of the lottery ticket hypothesis (Frankle and Carbin, 2019), looking specifically at the ‘‘winning’’ subnetworks in pre-trained BERT.”
- Revealing the Dark Secrets of BERT (2019, background): “Frankle and Carbin (2018) showed that widely used complex architectures suffer from overparameterization, and can be significantly reduced in size without a loss in performance.”
- What is the State of Neural Network Pruning? (2020, methodology): “Others consider scores globally, comparing scores to one another irrespective of the part of the network in which the parameter resides (Lee et al., 2019b; Frankle & Carbin, 2019).”
- Weight Agnostic Neural Networks (2019, background): “By retaining the original weight initialization values, these sparse networks can even be trained from scratch to achieve a higher test accuracy [22] than the original network.”
- Dynamic Model Pruning with Feedback (2020, methodology): “Note that Frankle & Carbin (2019); Frankle et al. (2019) do not propose an efficient pruning scheme to find the mask, instead they rely on iterative pruning, repeated for several full training passes.”
- Differentiable Neural Input Search for Recommender Systems (2020, positive): “Network pruning, as one kind of model compression techniques, improves the efficiency of over-parameterized deep neural networks by removing redundant neurons or connections without damaging model performance [7, 14, 26].”
- Fast Lossless Neural Compression with Integer-Only Discrete Flows (2022, background): “…network to convergence, scores the weight parameters (usually based on l 1 /l 2 -norms), prunes out low-scored parameters, and fine-tunes the pruned networks (LeCun et al., 1990; Han et al., 2015; Li et al., 2017; Lebedev & Lempitsky, 2016; Wen et al., 2016; He et al., 2017; Frankle & Carbin, 2019).”