On Warm-Starting Neural Network Training

266 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, neutral): “The same phenomenon is also observed in recent works [3].”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, neutral): “It could be attributed to the loss of plasticity, similar to the effect observed in [5].”
Continually Learning Self-Supervised Representations with Projected Functional Regularization (2021, background): “This is provides a tighter upper bound than Single Task [3].”
Loss of plasticity in deep continual learning (2024, unknown): “Plasticity loss with modern methods was visible in some recent works 11,16–18 and most recently has begun to be explored explicitly 12,19–27 .”
TiC-CLIP: Continual Training of CLIP Models (2023, unknown): “As such, Restart helps us understand the forward transfer and loss of plasticity in our benchmark (Ash & Adams, 2020; Dohare et al., 2023).”
Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning (2024, methodology): “…rule is given by w l,i,j ← w l,i,j − α [ ∂ L / ∂w l,i,j + ξ ] . as and It has been shown that a PGD with weight decay algorithm, known Shrink Perturb (S&P) (Ash & Adams 2020), can help maintain plasticity in continual classification problems (Dohare et al. 2023a) since maintaining small weights…”