Stabilizing the Lottery Ticket Hypothesis

By Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin. 2019

256 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “[29] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin.”
Dynamic Model Pruning with Feedback (2020, methodology): “Note that Frankle & Carbin (2019); Frankle et al. (2019) do not propose an efﬁcient pruning scheme to ﬁnd the mask, instead they rely on iterative pruning, repeated for several full training passes.”
Pruning neural networks without any data by iteratively conserving synaptic flow (2020, background): “Typically, it requires significantly more computational costs to identify winning tickets through iterative training and pruning cycles than simply training the original network from scratch [10, 11].”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, background): “…proposed to use one generalized lottery tickets for all vision benchmarks and obtained comparable results with the specialized lottery tickets; Frankle et al. (2019) improves the stability of the lottery tickets by iterative pruning; You et al. (2019); Chen et al. (2021) attempt to make it…”
The Early Phase of Neural Network Training (2020, methodology): “We perform extensive measurements of the network state and its updates during these early iterations of training, and leverage the framework of Frankle et al. (2019) to quantitatively probe the weight distribution and its reliance on various aspects of the dataset.”
How Well Do Sparse ImageNet Models Transfer? (2021, methodology): “The final category comprises Lottery Ticket Hypothesis (LTH)-style methods [5, 6, 14, 15].”
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win (2020, methodology): “(b) Transfer of LTs.”
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models (2021, background): “Many insightful studies [Morcos et al., 2019, Orseau et al., 2020, Frankle et al., 2019, 2020, Malach et al., 2020, Pensia et al., 2020] are carried out to analyze these tickets, but it remains difficult to generalize to large models due to training cost.”
Evaluating Lottery Tickets Under Distributional Shifts (2019, methodology): “Iterative magnitude pruning (Frankle and Carbin, 2019; Frankle et al., 2019) is a recently proposed method for finding small, sparse subnetworks from large, over-parameterized neural networks that can be trained in isolation to reach a similar (or better) test accuracy.”
Picking Winning Tickets Before Training by Preserving Gradient Flow (2020, methodology): “We further experiment with the late resetting as suggested in Frankle et al. (2019) by resetting the weights of the winning tickets to their values at 5 epoch.”
A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “…either modifications to the training procedure (Sanh et al., 2020; Kusupati et al., 2020), retraining the pruned networks to regain accuracy (Liu et al., 2019; Zhou et al., 2023), or an even more computationally intensive iterative retraining process (Renda et al., 2020; Frankle et al., 2020).”
Winning the Lottery with Continuous Sparsification (2019, methodology): “The proposal of Iterative Magnitude Pruning (IMP; Algorithm 1) [19] supports this hypothesis.”