One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers

By Ari S. Morcos, Haonan Yu, Michela Paganini, Yuandong Tian. 2019

256 citations recorded by the source corpus.

The success of lottery ticket initializations (Frankle and Carbin, 2019) suggests that small, sparsified networks can be trained so long as the network is initialized appropriately. Unfortunately, finding these "winning ticket" initializations is computationally expensive. One potential solution is to reuse the same winning tickets across a variety of datasets and optimizers. However, the generality of winning ticket initializations remains unclear. Here, we attempt to answer this question by generating winning tickets for one training configuration (optimizer and dataset) and evaluating their performance on another configuration. Perhaps surprisingly, we found that, within the natural images domain, winning ticket initializations generalized across a variety of datasets, including Fashion MNIST, SVHN, CIFAR-10/100, ImageNet, and Places365, often achieving performance close to that of winning tickets generated on the same dataset. Moreover, winning tickets generated using larger datasets consistently transferred better than those generated using smaller datasets. We also found that winning ticket initializations generalize across optimizers with high performance. These results suggest that winning ticket initializations generated by sufficiently large datasets contain inductive biases generic to neural networks more broadly which improve training across many settings and provide hope for the development of better initialization methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

What is the State of Neural Network Pruning? (2020, background): “Interestingly, this does not always hold for small amounts of pruning (Morcos et al., 2019).”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, methodology): “Our technique has already been adopted to create trainable subnetworks that transfer to new settings (Morcos et al., 2019), as a pruning method in its own right (Anonymous, 2020a), and to further study the lottery ticket hypothesis (Anonymous, 2020e;c;g;f;b).”
Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning (2020, methodology): “14 We might consider ﬁnding a lottery ticket for BERT, which we would expect to ﬁt the GLUE training data just as well as pre-trained BERT (Morcos et al., 2019; Yu et al., 2019).”
When BERT Plays the Lottery, All Tickets Are Winning (2020, background): “The “winning” initializations were shown to generalize across computer vision datasets (Morcos et al., 2019), and to exist both in LSTM and Transformer models for NLP tasks (Yu et al.”
The Lottery Ticket Hypothesis for Pre-trained BERT Networks (2020, background): “We have reason to believe that such transfer is possible: Morcos et al. [21] and Mehta [22] show that matching subnetworks transfer between vision tasks.”
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win (2020, background): “Morcos et al. (2019); Sabatelli, Kestemont, and Geurts (2020) showed that LTs trained on large datasets transfer to smaller ones, but not vice versa.”
TX-Ray: Quantifying and Explaining Model-Knowledge Transfer in (Un-)Supervised NLP (2019, background): “2a), which helps to decide which ‘tickets’ (components) generalize and transfer model-knowledge [25] or specialize it.”
Pruning neural networks without any data by iteratively conserving synaptic flow (2020, background): “Iterative Magnitude Pruning (IMP) is a recently proposed pruning algorithm that has proven to be successful in finding extremely sparse trainable neural networks at initialization (winning lottery tickets) [10, 11, 12, 44, 45, 46, 47].”
Winning the Lottery with Continuous Sparsification (2019, background): “Moreover, they can be transferred between datasets [16, 17] and training methods [15].”
Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning (2021, background): “Following that, a few works (Morcos et al., 2019; Mehta, 2019) have explored LTH in transfer learning.”
Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot (2020, methodology): “So it is not surprising to see that Figure A1 in [32] show evidence that the method may pass the layerwise rearrange check.”
TabTransformer: Tabular Data Modeling Using Contextual Embeddings (2020, methodology): “…the performance of TabTransformer against following four categories of methods: (a) Logistic regression and GBDT (b) MLP and a sparse MLP following (Morcos et al. 2019) (c) TabNet model of Arik and Pfister (2019) (d) and the Variational Information Bottleneck model (VIB) of Alemi et al. (2017).”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, result): “[75] proposed to use one generalized lottery tickets for all vision benchmarks and got comparable results with the specialized lottery tickets; Frankle et al.”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “Tuli and Jha [152] introduce DynaTran, which prunes activations at runtime based on the magnitude of the input matrix to enhance transformer inference throughput.”
Evaluating Lottery Tickets Under Distributional Shifts (2019, background): “Morcos et al. (2019) investigate the transferability of lottery tickets across multiple optimizers and datasets for supervised image classification, showing that tickets can indeed generalize (Morcos et al., 2019).”
Layer-adaptive Sparsity for the Magnitude-based Pruning (2020, methodology): “5, we plot the layerwise survival rates and the number of nonzero weights discovered by iteratively pruning VGG-16 (trained on CIFAR-10), by Global MP, Erd˝os-R´enyi kernel, and LAMP.”
Model Pruning Enables Efficient Federated Learning on Edge Devices (2019, methodology): “A finding that suggests a network that is pruned by magnitude consists of an optimal substructure of the original network, known as “lottery ticket hypothesis”, was presented in [11], [14].”
Drawing early-bird tickets: Towards more efficient training of deep networks (2019, background): “However, finding winning tickets hinged on costly (iterative) pruning and retraining (Morcos et al., 2019) studies the reuse of winning tickets, transferable across different datasets.”