Overcoming Multi-Model Forgetting

By Yassine Benyahia, Kaicheng Yu, Kamil Bennani-Smires, Martin Jaggi, A. C. Davison, Mathieu Salzmann, Claudiu Cristian Musat. 2019

40 citations recorded by the source corpus.

We identify a phenomenon, which we refer to as multi-model forgetting, that occurs when sequentially training multiple deep networks with partially-shared parameters; the performance of previously-trained models degrades as one optimizes a subsequent one, due to the overwriting of shared parameters. To overcome this, we introduce a statistically-justified weight plasticity loss that regularizes the learning of a model's shared parameters according to their importance for the previous models, and demonstrate its effectiveness when training two models sequentially and for neural architecture search. Adding weight plasticity in neural architecture search preserves the best models to the end of the search and yields improved results in both natural language processing and computer vision tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiaojun Chang
2. Shirui Pan
3. Miao Zhang
4. Steven W. Su
5. Chuan Zhou
6. Fei Mi
7. Reza Haffari
8. Tao Lin

In Computer Vision and Pattern Recognition

1. Shirui Pan
2. Miao Zhang
3. Steven W. Su
4. Xiaojun Chang
5. Ehsan Abbasnejad
6. Reza Haffari
7. Tao Lin
8. Boi Faltings

In Meta & Continual Learning

1. Fei Mi
2. Reza Haffari
3. Xiaojun Chang
4. Tao Lin
5. Steven W. Su
6. Boi Faltings
7. Lingjing Kong
8. Miao Zhang

Citations in the corpus

• One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, methodology): “WPL [5] regularizes the learning of current architecture by maximizing the posterior probability pðua; upb ; u j DÞ, where ua 1⁄4 fua; usg is the weights of the last architecture, ub 1⁄4 fub ; usg is the weights of the current architecture, and u is their shared weights.”
• Overcoming Multi-Model Forgetting in One-Shot NAS With Diversity Maximization (2020, methodology): “Different from jointly optimizing the posterior probability as WPL [5] under the assumption that θ a is near-optimal or keeping the shared weights fixed as Learn to Grow [21], we formulate the supernet training in the One-Shot NAS as a constrained optimization problem.”
• iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, background): “Different from the majority of existing works that attributes the failure of DARTS to its supernet optimization (Zela et al., 2020a; Benyahia et al., 2019), or the final discretization with argmax (Chen & Hsieh, 2020; Wang et al., 2021b), this paper revisits DARTS from the perspective of the…”
• Generalized Class Incremental Learning (2020, background): “Regularization [16, 14, 25, 4] and Ex-emplar Replay [21, 7, 5] are two major lines of research targeted at mitigating catastrophic forgetting.”