Wide Neural Networks Forget Less Catastrophically

By Seyed Iman Mirzadeh, Arslan Chaudhry, Huiyi Hu, Razvan Pascanu, Dilan Gorur, Mehrdad Farajtabar. 2021

88 citations recorded by the source corpus.

A primary focus area in continual learning research is alleviating the"catastrophic forgetting"problem in neural networks by designing new algorithms that are more robust to the distribution shifts. While the recent progress in continual learning literature is encouraging, our understanding of what properties of neural networks contribute to catastrophic forgetting is still limited. To address this, instead of focusing on continual learning algorithms, in this work, we focus on the model itself and study the impact of"width"of the neural network architecture on catastrophic forgetting, and show that width has a surprisingly significant effect on forgetting. To explain this effect, we study the learning dynamics of the network from various perspectives such as gradient orthogonality, sparsity, and lazy training regime. We provide potential explanations that are consistent with the empirical results across different architectures and continual learning benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Model Architectures, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Model Architectures

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “The third comes down to obtaining well-distributed representations, which tend to be more robust to distribution differences in function space, such as by using large-scale pre-training [87], [89], [90] and self-supervised learning [28], [91], [92], [93].”
The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “This principled way of increasing the network width complements the empirical claim of Mirzadeh et al. (2022) and the experimental observation of Rusu et al. (2016); Yoon et al. (2018): Wide neural networks forget less catastrophically.”
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, positive): “Recent work observed that a wider network can suffer from less catastrophic forgetting [28], while different components such as batch normalization, skip connections and pooling layers play various roles [29].”
Architecture Matters in Continual Learning (2022, background): “Recently, Mirzadeh et al. (2021a) have shown that wide neural networks forget less catastrophically by studying the impact of width and depth in MLP and WideResNet models.”
How catastrophic can catastrophic forgetting be in linear regression? (2022, background): “[42] Seyed Iman Mirzadeh, Arslan Chaudhry, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar.”
Scaling the Number of Tasks in Continual Learning (2022, methodology): “We also evaluate models with increasing wideness similarly to Mirzadeh et al. [2021] to estimate if wider models are less sensitive to distribution shifts.”
Efficient Continual Learning Ensembles in Neural Network Subspaces (2022, methodology): “In the context of continual learning, (Mirzadeh et al., 2021b) have recently shown that multitask and continual minima are connected via low-loss paths and leveraged this property to connect tasks’ minima in continual learning and proposed the MC-SGD algorithm, which encourages the linear…”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, background): “There is also a growing body of work demonstrating that model scale and dataset scale can make models more resistant to forgetting [34, 35], as well as work characterizing how forgetting naturally occurs in image classifiers [36].”