The Lottery Ticket Hypothesis at Scale

By Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin. 2019

105 citations recorded by the source corpus.

Recent work on the"lottery ticket hypothesis"proposes that randomly-initialized, dense neural networks contain much smaller, fortuitously initialized subnetworks ("winning tickets") capable of training to similar accuracy as the original network at a similar speed. While strong evidence exists for the hypothesis across many settings, it has not yet been evaluated on large, state-of-the-art networks and there is even evidence against the hypothesis on deeper networks. We modify the lottery ticket pruning procedure to make it possible to identify winning tickets on deeper networks. Rather than set the weights of a winning ticket to their original initializations, we set them to the weights obtained after a small number of training iterations ("late resetting"). Using late resetting, we identify the first winning tickets for Resnet-50 on Imagenet To understand the efficacy of late resetting, we study the"stability"of neural network training to pruning, which we define as the consistency of the optimization trajectories followed by a winning ticket when it is trained in isolation and as part of the larger network. We find that later resetting produces stabler winning tickets and that improved stability correlates with higher winning ticket accuracy. This analysis offers new insights into the lottery ticket hypothesis and the dynamics of neural network learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

What is the State of Neural Network Pruning? (2020, methodology): “Alternative proposals include rewinding the network to an earlier state (Frankle et al., 2019) and reinitializing the network entirely (Liu et al.”
Rigging the Lottery: Making All Tickets Winners (2019, background): “Frankle et al. (2019) showed that the formulation must be weakened to apply to larger networks such as ResNet-50 (He et al., 2015).”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, background): “Interpretation and Analysis of Sparse Neural Networks : Frankle and Carbin (2019) show that “winning lottery tickets” exist for deep neural networks – sparse initializations which reach similar predictive performance as dense networks and train just as fast.”
The neural architecture of language: Integrative modeling converges on predictive processing (2020, background): “These ideas have successfully been applied to kernel machines (Rahimi & Recht, 2009) and are more recently explored again with deep neural networks (Frankle et al., 2019); in short, it is possible that even random features with the right multiscale structure in time and space could be more powerful…”
NeuralScale: Efficient Scaling of Neurons for Resource-Constrained Deep Neural Networks (2020, methodology): “The idea of using pruning as an architecture search method is not novel and has been discussed in [33, 9, 10, 7] where its applicability can be seen in MorphNet [12].”
Picking Winning Tickets Before Training by Preserving Gradient Flow (2020, methodology): “We further experiment with the late resetting as suggested in Frankle et al. (2019) by resetting the weights of the winning tickets to their values at 5 epoch.”
Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning (2021, background): “(Frankle et al., 2019) analyzes large models and datasets, and presents a rewinding technique that re-initializes ticket training from the early training stage rather than from scratch.”
One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers (2019, background): “Consistent with previous results [8], performance on large-scale datasets began to rapidly degrade for ResNet50 models when approximately 5-10% of weights remained, in contrast to VGG19 models which only demonstrated small decreases in accuracy, even with 99.”