Improving One-Shot NAS by Suppressing the Posterior Fading

By Xiang Li, Chen Lin, Chuming Li, Ming Sun, Wei Wu, Junjie Yan, Wanli Ouyang. 2019

80 citations recorded by the source corpus.

Neural architecture search (NAS) has demonstrated much success in automatically designing effective neural network architectures. To improve the efficiency of NAS, previous approaches adopt weight sharing method to force all models share the same set of weights. However, it has been observed that a model performing better with shared weights does not necessarily perform better when trained alone. In this paper, we analyse existing weight sharing one-shot NAS approaches from a Bayesian point of view and identify the Posterior Fading problem, which compromises the effectiveness of shared weights. To alleviate this problem, we present a novel approach to guide the parameter posterior towards its true distribution. Moreover, a hard latency constraint is introduced during the search so that the desired latency can be achieved. The resulted method, namely Posterior Convergent NAS (PC-NAS), achieves state-of-the-art performance under standard GPU latency constraint on ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiaojun Chang
2. Xiaodan Liang
3. Junjie Yan
4. Po-Yao (Bernie) Huang
5. Zhihui Li
6. Shirui Pan
7. Wanli Ouyang
8. Chuan Zhou

In Computer Vision and Pattern Recognition

1. Xiaodan Liang
2. Wanli Ouyang
3. Xiaogang Wang
4. Xiaojun Chang
5. Junjie Yan
6. Luc Van Gool
7. Yi Yang
8. Dacheng Tao

Citations in the corpus

• A Comprehensive Survey of Neural Architecture Search (2020, background): “In addition, the research of Fairnas [100] and [101] also shows that candidate network architectures based on these parameter sharing often cannot be adequately trained, which will lead to inaccurate ranking of candidate network architectures.”
• BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, background): “1a, architecture search spaces with layer-level granularity grow exponentially with increased network depth, which has been identified (in [36, 39]) as the main culprit of inaccurate architecture rating2 in weight-sharing NAS methods.”
• GLiT: Neural Architecture Search for Global and Local Image Transformer (2021, methodology): “Recently, NAS methods make great progress on the vision tasks [5, 9, 23, 35, 20, 21].”
• Overcoming Multi-Model Forgetting in One-Shot NAS With Diversity Maximization (2020, background): “It is also observed in [30] and [20] that the catastrophic forgetting in the One-Shot NAS would deteriorate the performance of previous architectures after training a new architecture in the supernet.”
• AMER: Automatic Behavior Modeling and Interaction Exploration in Recommender System (2020, methodology): “…model and child networks, AMER modularizes the searching process into three steps matching the three stages in the backbone model according to [38, 61], meanwhile ensuring the adequacy and fairness of one-shot model-training to alleviate representation shift caused by weight sharing [10, 79].”
• One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, background): “Several recent studies [31], [53], [61] have shown that the interactions between networks can degrade the performance of a whole network, and that catastrophic forgetting with one-shot NAS can lower the performance of previous architectures after training a new”