Critical Learning Periods in Deep Neural Networks

By Alessandro Achille, Matteo Rovere, Stefano Soatto. 2017

124 citations recorded by the source corpus.

Similar to humans and animals, deep artificial neural networks exhibit critical periods during which a temporary stimulus deficit can impair the development of a skill. The extent of the impairment depends on the onset and length of the deficit window, as in animal models, and on the size of the neural network. Deficits that do not affect low-level statistics, such as vertical flipping of the images, have no lasting effect on performance and can be overcome with further training. To better understand this phenomenon, we use the Fisher Information of the weights to measure the effective connectivity between layers of a network during training. Counterintuitively, information rises rapidly in the early phases of training, and then decreases, preventing redistribution of information resources in a phenomenon we refer to as a loss of "Information Plasticity". Our analysis suggests that the first few epochs are critical for the creation of strong connections that are optimal relative to the input data distribution. Once such strong connections are created, they do not appear to change during additional training. These findings suggest that the initial learning transient, under-scrutinized compared to asymptotic behavior, plays a key role in determining the outcome of the training process. Our findings, combined with recent theoretical results in the literature, also suggest that forgetting (decrease of information in the weights) is critical to achieving invariance and disentanglement in representation learning. Finally, critical periods are not restricted to biological systems, but can emerge naturally in learning systems, whether biological or artificial, due to fundamental constrains arising from learning dynamics and information processing.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Optimization & Training

In Scaling & Learning Theory

Citations in the corpus

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, background): “Achille et al. [2017] showed the importance of early learning from the perspective of creating strong connections that are robust to corruption.”
Continual Lifelong Learning with Neural Networks: A Review (2018, background): “Recent studies on critical periods in deep neural networks showed that the initial rapid learning phase plays a key role in defining the final performance of the networks (Achille et al. 2017).”
Triple-Memory Networks: A Brain-Inspired Method for Continual Learning (2020, background): “Particularly, our weight regularization process keeps biologically plausible because empirical FIM is also a measurement of synaptic connectivity of a network Achille et al. [2017].”
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks (2019, background): “Forgetting without the pretraining assumption is an interesting challenge, as it has been observed that slight perturbation of the initial critical learning period can lead to large difference in the final solution [2, 12].”
Task2Vec: Task Embedding for Meta-Learning (2019, background): “FIM has also been proposed for various regularization schemes [5, 8, 22, 27], to analyze learning dynamics of deep networks [4], and to overcome catastrophic forgetting [19].”
Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias (2019, background): “First, the sharpness (|∇L|) and steepness (λmax) indicators increase then decrease during the training of the CNN (as analyzed in [1]), and display a maximum around tw ' 20, which coincides with the relax time of best improvement for the eFCNs.”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “Leveraging training signal There are several prior techniques that also leverage network training dynamics over distinct phases of learning (Achille et al., 2017; Jiang et al., 2020; Mangalam and Prabhu, 2019; Faghri et al., 2020; Agarwal et al., 2021) are challenging yet learnable.”
The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, methodology): “It has been observed that applying regularization in the early phase of training is necessary to arrive at a well generalizing ﬁnal solu-tion (Keskar et al., 2017; Sagun et al., 2017; Achille et al., 2017).”
Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data (2019, result): “Recent works [1, 16] found that regularization does not need to exist in the whole training process, which is compatible with our work.”
The Information Complexity of Learning Tasks, their Structure and their Distance (2019, background): “The existence of tasks that are similar according to most distances, yet one cannot “reach” one from another, has been observed empirically in [3].”
Heterogeneous Knowledge Distillation Using Information Flow Modeling (2020, methodology): “However, as argued in [1], the information plasticity rapidly declines after the first few training epochs, reducing the effectiveness of knowledge distillation.”
The Dynamics of Differential Learning I: Information-Dynamics and Task Reachability (2018, background): “When are two tasks “close”? Can one measure the distance between tasks without actually running an experiment? Does knowing this distance help predict whether transfer learning is possible, and if so how many resources or time will be needed? Surely the distance between tasks is not just the lexicog”
The Early Phase of Neural Network Training (2020, methodology): “ training. For example, sparse, trainable sub-networks emerge (Frankle et al., 2019), gradient descent moves into a small subspace (Gur-Ari et al., 2018), and the network undergoes a critical period (Achille et al., 2019). Here we examine the changes that deep neural networks undergo during this ear”
Pruning neural networks: is it time to nip it in the bud? (2018, unknown): “, 2017) or the importance of the early phases of learning (Jastrzębski et al., 2018b; Achille et al., 2017), necessitating that the network should be trained from scratch rather than fine-tuned (He et al.”