Critical Learning Periods in Deep Networks

By Alessandro Achille, Matteo Rovere, Stefano Soatto. 2018

169 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

Citations in the corpus

DARTS+: Improved Differentiable Architecture Search with Early Stopping (2019, background): “It has been found that important connections are determined in the early phase of training [1].”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…has been shown to speed up the training of neural networks, while also leading to better generalization (Bengio et al., 2009; Graves, Bellemare, Menick, Munos, & Kavukcuoglu, 2017; Hunziker et al., 2019; Zhou & Bilmes, 2018; Fan, Tian, Qin, Li, & Liu, 2018; Achille, Rovere, & Soatto, 2018).”
Where is the Information in a Deep Neural Network? (2019, background): “ the Fisher matrix directly from the deﬁnition. As different features are learned, we observe corresponding “bumps” in the Fisher Information matrix. This is compatible with the hypothesis advanced byAchille et al. (2019), whereby feature learning may correspond to crossing a narrow bottleneck (high”
The Dynamics of Differential Learning I: Information-Dynamics and Task Reachability (2018, background): “depends on the architecture). Moreover, this suggests that the dynamic part of the potential can create spurious local minima that can inhibit learning of new problems in a transfer learning scenario [3]. However,eq. (14)still depends on the geometry of the optimization landscape, rather than proper”
FDA: Fourier Domain Adaptation for Semantic Segmentation (2020, background): “This is especially important since it appears that networks do not transfer well across different low-level statistics [1].”
On Warm-Starting Neural Network Training (2019, background): “For example, in analyzing “critical learning periods,” researchers show that a network initially trained on blurry images then on sharp images is unable to perform as well as one trained from scratch on sharp images, drawing a parallel between human vision and computer vision [26].”
Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence (2019, methodology): “We use the methodology of [1] to regress the most critical epochs for various architectures and datasets.”
TiC-CLIP: Continual Training of CLIP Models (2023, unknown): “7 ˆ less compute. initiating training from a pre-existing model can make it difficult to change the model’s behavior in light of new data (Ash & Adams, 2020; Achille et al., 2018; Liu et al., 2023).”
Drawing early-bird tickets: Towards more efficient training of deep networks (2019, unknown): “An important hint can be found in (Achille et al., 2019): the early stage of training seems to first discover the important connections and the connectivity patterns between layers, which becomes relatively fixed in the later training stage.”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, background): “Language Model Training Dynamics: Previous work has extensively analyzed training dynamics to understand how neural models acquire information over training [1, 30, 34, 66, 74].”
CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery (2020, background): “, impaired input image) is more harmful at an early stage of training (Achille et al., 2018).”