The Early Phase of Neural Network Training

By Jonathan Frankle, David J. Schwab, Ari S. Morcos. 2020

202 citations recorded by the source corpus.

Recent studies have shown that many important aspects of neural network learning take place within the very earliest iterations or epochs of training. For example, sparse, trainable sub-networks emerge (Frankle et al., 2019), gradient descent moves into a small subspace (Gur-Ari et al., 2018), and the network undergoes a critical period (Achille et al., 2019). Here we examine the changes that deep neural networks undergo during this early phase of training. We perform extensive measurements of the network state and its updates during these early iterations of training, and leverage the framework of Frankle et al. (2019) to quantitatively probe the weight distribution and its reliance on various aspects of the dataset. We find that, within this framework, deep networks are not robust to reinitializing with random weights while maintaining signs, and that weight distributions are highly non-independent even after only a few hundred iterations. Despite this, pre-training with blurred inputs or an auxiliary self-supervised task can approximate the changes in supervised networks, suggesting that these changes are label-agnostic, though labels significantly accelerate this process. Together, these results help to elucidate the network changes occurring during this pivotal initial period of learning.

Subfields: Artificial Intelligence, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

Citations in the corpus

The large learning rate phase of deep learning: the catapult mechanism (2020, background): “SGD training with large initial learning rates often leads to improved performance over training with small initial learning rates (see Li et al. (2019); Leclerc & Madry (2020); Xie et al. (2020); Frankle et al. (2020); Jastrzebski et al. (2020) for recent discussions).”
DARTS+: Improved Differentiable Architecture Search with Early Stopping (2019, background): “Significant and consequential changes occur during the earliest stage of training [7].”
Understanding the Role of Training Regimes in Continual Learning (2020, background): “The learning rate and batch size inﬂuence both the endpoint curvature and the whole trajectory [73, 16, 43].”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, background): “Other efforts use rewinding to further study lottery tickets (Yu et al., 2020; Frankle et al., 2020; Caron et al., 2020; Savarese et al., 2020; Yin et al., 2020).”
The Lottery Ticket Hypothesis for Pre-trained BERT Networks (2020, methodology): “We therefore explore another form of random initialization: shufﬂing the BERT pre-trained weights within each layer to obtain a new initialization θ 0 [19].”
Drawing early-bird tickets: Towards more efficient training of deep networks (2019, unknown): “…the lottery ticket training in (Frankle & Carbin, 2019), EB Train inherits the unpruned weights from the drawn EB ticket instead of rewinding to the original initialization, as it has been shown that deeper networks are not robust to reinitializion with untrained weights (Frankle et al., 2020).”
Playing Lottery Tickets with Vision and Language (2021, background): “The properties of LTH are widely studied for image classification [44, 15, 18, 58, 68, 74, 47], and recently, the observations of LTH have also been evidenced across other fields, such as natural language processing [19, 76, 54, 6, 9], object detection [21], generative adversarial networks [10, 31, ”
Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning (2021, background): “Moreover, the introduction of weight rewinding is spurred by the so-called rewinding ticket (Renda et al., 2020; Frankle et al., 2020).”
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win (2020, methodology): “…different solutions is a popular tool for understanding how various points in parameter space relate to each other in the loss landscape (Goodfellow, Vinyals, and Saxe 2015; Garipov et al. 2018; Draxler et al. 2018; Evci et al. 2019; Fort, Hu, and Lakshminarayanan 2020; Frankle et al. 2020a).”
Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture (2020, result): “We also recommend looking into why CLT pruning works in our setup, while Frankle et al. (2020) present opposing results.”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, unknown): “Language Model Training Dynamics : Previous work has extensively analyzed training dynamics to understand how neural models acquire information over training [1, 30, 34, 66, 74].”