Overcoming Multi-Model Forgetting in One-Shot NAS With Diversity Maximization

By Miao Zhang, Huiqi Li, Shirui Pan, Xiaojun Chang, Steven W. Su. 2020

89 citations recorded by the source corpus.

One-Shot Neural Architecture Search (NAS) significantly improves the computational efficiency through weight sharing. However, this approach also introduces multi-model forgetting during the supernet training (architecture search phase), where the performance of previous architectures degrade when sequentially training new architectures with partially-shared weights. To overcome such catastrophic forgetting, the state-of-the-art method assumes that the shared weights are optimal when jointly optimizing a posterior probability. However, this strict assumption is not necessarily held for One-Shot NAS in practice. In this paper, we formulate the supernet training in the One-Shot NAS as a constrained optimization problem of continual learning that the learning of current architecture should not degrade the performance of previous architectures during the supernet training. We propose a Novelty Search based Architecture Selection (\textbf{NSAS}) loss function and demonstrate that the posterior probability could be calculated without the strict assumption when maximizing the diversity of the selected constraints. A greedy novelty search method is devised to find the most representative subset to regularize the supernet training. We apply our proposed approach to two One-Shot NAS baselines, random sampling NAS (RandomNAS) and gradient-based sampling NAS (GDAS). Extensive experiments demonstrate that our method enhances the predictive ability of the supernet in One-Shot NAS and achieves remarkable performance on CIFAR-10, CIFAR-100, and PTB with efficiency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Comprehensive Survey of Neural Architecture Search (2020, methodology): “But more and more evidence shows that the parameter sharing strategy is likely to result in a sub-optimal inaccurate candidate architecture ranking [21, 24, 25, 99].”
Dynamic Slimmable Network (2021, unknown): “…efﬁciency of neural networks, including network pruning [43, 50, 21, 22, 51, 53], weight quantization [33], knowledge distillation [2, 56, 23], manually and automatically designing of efﬁcient networks [60, 57, 77, 55, 3, 75, 10, 18, 41, 76] and dynamic inference [4, 30, 64, 63, 44, 29, 14].”
One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, background): “This paper outlines some significant extensions to our recent conference paper [65].”
Edge YOLO: Real-Time Intelligent Object Detection System Based on Edge-Cloud Cooperation in Autonomous Vehicles (2022, methodology): “(3) Using Neural Architecture Search (NAS) [33] technology in AutoDL [34] to design the Neural Network automatically, the high-efﬁciency Network Architectures can be obtained under the constraints of model size, number of parameters and other strategies.”
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, methodology): “On this challenging search space, our searched hybrid CNN-transformer model, achieves 82 [92, 2, 54, 63, 42, 83] rate all sampled architectures by training them from scratch, making this process computationally prohibitive and difﬁcult to deploy on large datasets.”
iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, background): “…more current works observe that DARTS is somewhat unreliable (Zela et al., 2020a; Chen & Hsieh, 2020; Li & Talwalkar, 2019; Sciuto et al., 2019; Zhang et al., 2020c; Li et al., 2021a; Zhang et al., 2020b) since it does not consistently yield excellent solutions, performing even worse than…”