Block-Wisely Supervised Neural Architecture Search With Knowledge Distillation

By Changlin Li, Jiefeng Peng, Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, Xiaojun Chang. 2019

198 citations recorded by the source corpus.

Neural Architecture Search (NAS), aiming at automatically designing network architectures by machines, is expected to bring about a new revolution in machine learning. Despite these high expectation, the effectiveness and efficiency of existing NAS solutions are unclear, with some recent works going so far as to suggest that many existing NAS solutions are no better than random architecture selection. The ineffectiveness of NAS solutions may be attributed to inaccurate architecture evaluation. Specifically, to speed up NAS, recent works have proposed under-training different candidate architectures in a large search space concurrently by using shared network parameters; however, this has resulted in incorrect architecture ratings and furthered the ineffectiveness of NAS. In this work, we propose to modularize the large search space of NAS into blocks to ensure that the potential candidate architectures are fully trained; this reduces the representation shift caused by the shared parameters and leads to the correct rating of the candidates. Thanks to the block-wise search, we can also evaluate all of the candidate architectures within each block. Moreover, we find that the knowledge of a network model lies not only in the network parameters but also in the network architecture. Therefore, we propose to distill the neural architecture (DNA) knowledge from a teacher model to supervise our block-wise architecture search, which significantly improves the effectiveness of NAS. Remarkably, the performance of our searched architectures has exceeded the teacher model, demonstrating the practicability of our method. Finally, our method achieves a state-of-the-art 78.4% top-1 accuracy on ImageNet in a mobile setting. All of our searched models along with the evaluation code are available at https://github.com/changlin31/DNA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Comprehensive Survey of Neural Architecture Search (2020, methodology): “In addition, DNA [21] also uses block-wise search to evaluate all candidate architectures within the block.”
Knowledge Distillation: A Survey (2020, methodology): “…including the design of knowledge (Hinton et al. 2015; Romero et al. 2015) and the loss functions for matching features or distributions matching (Huang and Wang 2017; Passalis and Tefas 2018; Zagoruyko and Komodakis 2017; Mirzadeh et al. 2020; Li et al. 2020d; Heo et al. 2019b; Asif et al. 2020).”
Towards Personalized Federated Learning (2021, background): “As it may be difficult for the student model to learn well if there is a huge capacity gap between the large teacher model and the small student model [73, 74], it is imperative to determine an optimal design for both the server and client models.”
Dynamic Slimmable Network (2021, unknown): “Weight sharing NAS [5, 1, 3, 48, 7, 66, 18, 41, 6, 42], aiming at designing neural network architectures automatically and efﬁciently, has been developing rapidly in recent two years.”
One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, background): “NEURAL architecture search (NAS) has re7cently attracted massive interest from the deep learning community because experts do not have inordinate amounts of time and labor designing neural networks [13], [18], [29], [35], [42], [47], [50], [71].”
iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, methodology): “…and effective method on automating the process of neural network design, with achieving remarkable success on image recognition (Tan & Le, 2019; Li et al., 2021b; 2020), language modeling (Jiang et al., 2019), and other deep learning ap-1 plications (Ren et al., 2020; Cheng et al., 2020; Chen…”
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, methodology): “By contrast, network-level search spaces with layer-level granularity [7, 70, 15, 37, 46, 87] and block-level granularity [63, 29, 64] search for the macro network-level structure us-ing manually designed building blocks ( e.g. MBConv [56]).”
Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search (2020, methodology): “Nevertheless, these methods require the teacher model to be trained beforehand, such as a large pretrained model [5] or a third-party model [6].”
Differentiable Feature Aggregation Search for Knowledge Distillation (2020, background): “DNA [20] searches for the light-weight architecture of the student network from a supernet.”