Architecture Matters in Continual Learning

By Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascanu, Dilan Gorur, Mehrdad Farajtabar. 2022

64 citations recorded by the source corpus.

A large body of research in continual learning is devoted to overcoming the catastrophic forgetting of neural networks by designing new algorithms that are robust to the distribution shifts. However, the majority of these works are strictly focused on the"algorithmic"part of continual learning for a"fixed neural network architecture", and the implications of using different architectures are mostly neglected. Even the few existing continual learning methods that modify the model assume a fixed architecture and aim to develop an algorithm that efficiently uses the model throughout the learning experience. However, in this work, we show that the choice of architecture can significantly impact the continual learning performance, and different architectures lead to different trade-offs between the ability to remember previous tasks and learning new ones. Moreover, we study the impact of various architectural decisions, and our findings entail best practices and recommendations that can improve the continual learning performance.

Subfields: Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “For example, wider networks tend to be more robust to catastrophic forgetting due to more orthogonality and sparsity in gradient directions, as well as a lazier training regime [305], [306].”
CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, background): “For example, one study varies the number of attention heads in Vision transformers and conclude that ViTs offer more robustness to forgetting with respect to CNN-based equivalents [44].”
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, positive): “Recent work observed that a wider network can suffer from less catastrophic forgetting [28], while different components such as batch normalization, skip connections and pooling layers play various roles [29].”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, neutral): “The observations we shown here are similar to what reported in a very recent study about how large architecture help continual learning [40].”
Wide Neural Networks Forget Less Catastrophically (2021, methodology): “We refer the reader to (Mirzadeh et al., 2022) for additional results on other benchmarks and architectures such as CNNs, ResNets (He et al., 2016), and Vision Transformers (Dosovitskiy et al., 2021).”
Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization (2022, background): “[32] also experiment with ViTs in brief (with the rest of the work focusing mainly on CNNs).”
Convolutional Prompting meets Language Models for Continual Learning (2024, background): “Authors in [33] examined the impact of attention heads while Dytox [12] acquires new skills by expanding special task tokens.”
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (2023, unknown): “Prior works have also analyzed different components in a LL agent [45, 70, 21], but they do not focus on robot manipulation problems.”