Linear Mode Connectivity in Multitask and Continual Learning

By Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, Hassan Ghasemzadeh. 2020

180 citations recorded by the source corpus.

Continual (sequential) training and multitask (simultaneous) training are often attempting to solve the same overall objective: to find a solution that performs well on all considered tasks. The main difference is in the training regimes, where continual learning can only have access to one task at a time, which for neural networks typically leads to catastrophic forgetting. That is, the solution found for a subsequent task does not perform well on the previous ones anymore. However, the relationship between the different minima that the two training regimes arrive at is not well understood. What sets them apart? Is there a local structure that could explain the difference in performance achieved by the two different schemes? Motivated by recent work showing that different minima of the same task are typically connected by very simple curves of low error, we investigate whether multitask and continual solutions are similarly connected. We empirically find that indeed such connectivity can be reliably achieved and, more interestingly, it can be done by a linear path, conditioned on having the same initialization for both. We thoroughly analyze this observation and discuss its significance for the continual learning process. Furthermore, we exploit this finding to propose an effective algorithm that constrains the sequentially learned minima to behave as the multitask solution. We show that our method outperforms several state of the art continual learning algorithms on various vision benchmarks.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Meta & Continual Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

In Statistical Learning Theory

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “MC-SGD [308] empirically demonstrates that the local minima obtained by multi-task learning (i.”
The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “This principled way of increasing the network width complements the empirical claim of Mirzadeh et al. (2022) and the experimental observation of Rusu et al. (2016); Yoon et al.”
Repeated Augmented Rehearsal: A Simple but Strong Baseline for Online Continual Learning (2022, methodology): “[2021], Mirzadeh et al. [2020], we examine the learning process on the first two tasks in the Split Mini-ImageNet dataset and plot the loss landscape in the 2D plane defined by three model parameter vectors2: the model w1 obtained by training on the first task until convergence, the model w2 obtaine”
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, positive): “They further exploited recent advances of mode connectivity [30] and neural network subspace [48] to save computational cost, but had to use old training samples [13].”
Continual Learning with Guarantees via Weight Interval Constraints (2022, background): “Finally, a recent work (Mirzadeh et al., 2021) addressed catastrophic forgetting by exploiting the linear connectivity between solutions obtained through multitask and continual learning in the form of Mode Connectivity SGD.”
Architecture Matters in Continual Learning (2022, background): “Recently, Mirzadeh et al. (2021a) have shown that wide neural networks forget less catastrophically by studying the impact of width and depth in MLP and WideResNet models.”
An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, background): “Finally, Mirzadeh et al. (2020a) compare minima that result from multitask learning and continual learning, and show that the the minima resulting from continual learning are linear mode connected to the optimal sequential multitask minima, but not to each other, which results in forgetting and a…”
Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, methodology): “To mitigate catastrophic forgetting, previous studies proposed several strategies such as regularization, parameter isolation, and experience replay (Mirzadeh et al., 2020).”
Wide Neural Networks Forget Less Catastrophically (2021, methodology): “In their seminal papers, Jacot et al. (2018) and Lee et al. (2019) connect learning in neural networks to kernel methods and show that at the inﬁnite width limit, the kernel doesn’t change throughout the training, and the neural network evolves as a linear model under SGD training.”
Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector (2021, neutral): “Although MC-SGD achieves excellent performance, it needs to store previous samples, which contradicts with our problem setting.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[183] pointed out that there exists a linear connection between continual learning and multi-task learning.”
Model Zoo: A Growing Brain That Learns Continually (2021, background): “Parameter sharing/isolation A single shared feature generator (i.e., hard parameter sharing) is a popular architecture (Kirkpatrick et al., 2017; Lopez-Paz and Ranzato, 2017; Rebuﬃ et al., 2017a; Nguyen et al., 2017; Mirzadeh et al., 2020b; Chaudhry et al., 2019b).”
Online Coreset Selection for Rehearsal-based Continual Learning (2021, positive): “We validate OCS on class-incremental CL for Balanced and Imbalanced Rotated MNIST using a single-head two-layer MLP with 256 ReLU units in each layer, task-incremental CL for Split CIFAR-100 and Multiple Datasets (a sequence of five datasets) with a multi-head structured ResNet18 following prior wor”
How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, methodology): “Moreover, we also conduct the visualization of relative loss change g over a linear path like (Mirzadeh et al., 2020) in Appendix C.4.”
Understanding the Role of Training Regimes in Continual Learning (2020, background): “[55] have studied the mode connectivity of continual learning and multitask learning minima.”
A Multi-Head Model for Continual Learning via Out-of-Distribution Replay (2022, background): “This approach approximates the importance of parameters for the previous tasks, and penalizes changes to them to prevent forgetting (Kirkpatrick et al., 2017; Zenke et al., 2017; Ritter et al., 2018; Dhar et al., 2019; Mirzadeh et al., 2021).”
Questions for Flat-Minima Optimization of Modern Neural Networks (2022, background): “…dynamics (Goodfellow & Vinyals, 2015; Frankle, 2020; Fort et al., 2020; Lucas et al., 2021; Yang et al., 2021), regularization (Li et al., 2018; Geiping et al., 2021), network pruning (Frankle et al., 2020), transfer learning (Neyshabur et al., 2020), or multitask learning (Mirzadeh et al., 2021).”
Task-agnostic Continual Learning with Hybrid Probabilistic Models (2021, unknown): “Modularity-based methods allocate diﬀerent subsets of the parameters to each task [Rusu et al., 2016, Yoon et al., 2018, Jerfel et al., 2019, Li et al., 2019, Wortsman et al., 2020, Mirzadeh et al., 2020a].”