Linear Mode Connectivity and the Lottery Ticket Hypothesis

By Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin. 2019

798 citations recorded by the source corpus.

We study whether a neural network optimizes to the same, linearly connected minimum under different samples of SGD noise (e.g., random data order and augmentation). We find that standard vision models become stable to SGD noise in this way early in training. From then on, the outcome of optimization is determined to a linearly connected region. We use this technique to study iterative magnitude pruning (IMP), the procedure used by work on the lottery ticket hypothesis to identify subnetworks that could have trained in isolation to full accuracy. We find that these subnetworks only reach full accuracy when they are stable to SGD noise, which either occurs at initialization for small-scale settings (MNIST) or early in training for large-scale settings (ResNet-50 and Inception-v3 on ImageNet).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “For model training, the lottery tickets hypothesis [30–32] suggests that there are a set of small sub-networks derived from a larger dense network that performs as well as the original dense network.”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “, 2019; Wilson and Izmailov, 2020) and network pruning (Frankle et al., 2020).”
Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, background): “However, finding “good” sparse sub-networks at initialization remains a central challenge for these techniques [13, 33].”
On the Predictability of Pruning Across Scales (2020, methodology): “For IMP, we use a practice called weight rewinding (Frankle et al., 2020; Renda et al., 2020), in which the values of unpruned weights are rewound to their values earlier in training (in our case, epoch 10) and the training process is repeated from there to completion.”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, background): “[32] find that, when training a pair of models from scratch with the same hyperparameter configuration but different data order, interpolating weights achieves no better than random accuracy.”
Linear Mode Connectivity in Multitask and Continual Learning (2020, methodology): “For a “single task”, common initialization leads to linear path between minima (Frankle et al., 2020).”
Movement Pruning: Adaptive Sparsity by Fine-Tuning (2020, background): “, 2019], and more recently has been leveraged as a core component in the lottery ticket hypothesis [Frankle et al., 2020].”
Robust fine-tuning of zero-shot models (2021, background): “Recall that we are ensembling quantities before a softmax is applied, so Figure 5: The zero-shot and ﬁne-tuned models exhibit linear mode connectivity (Frankle et al., 2020) on ImageNet and the main distribution shifts we consider (Observation 1).”
The Lottery Ticket Hypothesis for Pre-trained BERT Networks (2020, methodology): “To identify subnetworks f ( x ; m (cid:12) θ, · ) , we use neural network pruning [15, 16].”
Winning the Lottery with Continuous Sparsification (2019, background): “[14] show that although this approach can fail in large-scale settings, pruned networks can be successfully re-trained when parameters from very early training are used as initialization.”
Stabilizing the Lottery Ticket Hypothesis (2019, methodology): “, Gale et al. (2019), and Frankle and Carbin themselves show that IMP fails on deeper networks.”
Playing Lottery Tickets with Vision and Language (2021, methodology): “Though originally working only on small networks, later on, rewinding is found to be a useful technique to scale up LTH to large networks [57, 17].”
Editing Models with Task Arithmetic (2022, neutral): “[27] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin.”
Deep Learning on a Data Diet: Finding Important Examples Early in Training (2021, background): “In particular, Frankle et al. [22] studied the effect of minibatch randomness on the training trajectory, focusing on identifying the point in training when two networks, starting from the same weights, but trained with independent minibatches, converge to the same “linearly connected” mode.”
Learning Neural Network Subspaces (2021, methodology): “(Frankle et al., 2020; Fort et al., 2020) Ens-membling θ 1 T and θ 2 T in weight space—making predictions with the network f x , —fails, achieving no better accuracy than an untrained network.”
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks (2021, methodology): “Frankle et al. (2020) showed that solutions that are linearly connected with no barrier have the same lottery ticket.”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, positive): “…effective way to develop domain-speciﬁc language models (Gururangan et al., 2020), and parameter interpolation techniques work best with models that have a shared initialization (Izmailov et al., 2018; Frankle et al., 2020; Wortsman et al., 2022b; Matena and Raffel, 2021; Wortsman et al., 2022a) .”
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models (2021, background): “Many insightful studies [Morcos et al., 2019, Orseau et al., 2020, Frankle et al., 2019, 2020, Malach et al., 2020, Pensia et al., 2020] are carried out to analyze these tickets, but it remains difficult to generalize to large models due to training cost.”