Semi-Supervised Neural Architecture Search

By Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Enhong Chen, Tie-Yan Liu. 2020

104 citations recorded by the source corpus.

Neural architecture search (NAS) relies on a good controller to generate better architectures or predict the accuracy of given architectures. However, training the controller requires both abundant and high-quality pairs of architectures and their accuracy, while it is costly to evaluate an architecture and obtain its accuracy. In this paper, we propose SemiNAS, a semi-supervised NAS approach that leverages numerous unlabeled architectures (without evaluation and thus nearly no cost). Specifically, SemiNAS 1) trains an initial accuracy predictor with a small set of architecture-accuracy data pairs; 2) uses the trained accuracy predictor to predict the accuracy of large amount of architectures (without evaluation); and 3) adds the generated data pairs to the original data to further improve the predictor. The trained accuracy predictor can be applied to various NAS algorithms by predicting the accuracy of candidate architectures for them. SemiNAS has two advantages: 1) It reduces the computational cost under the same accuracy guarantee. On NASBench-101 benchmark dataset, it achieves comparable accuracy with gradient-based method while using only 1/7 architecture-accuracy pairs. 2) It achieves higher accuracy under the same computational cost. It achieves 94.02% test accuracy on NASBench-101, outperforming all the baselines when using the same number of architectures. On ImageNet, it achieves 23.5% top-1 error rate (under 600M FLOPS constraint) using 4 GPU-days for search. We further apply it to LJSpeech text to speech task and it achieves 97% intelligibility rate in the low-resource setting and 15% test error rate in the robustness setting, with 9%, 7% improvements over the baseline respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Xu Tan
3. Tie-Yan Liu
4. Sheng Zhao
5. Enhong Chen
6. Renqian Luo
7. Zhangyang Wang
8. Yi Ren

In Computer Vision and Pattern Recognition

1. Tie-Yan Liu
2. Tao Qin
3. Xu Tan
4. Zhangyang Wang
5. Wuyang Chen
6. Enhong Chen
7. Renqian Luo
8. Rui Wang

In Neural Architecture Search & AutoML

1. Wuyang Chen
2. Zhangyang Wang
3. Xinyu Gong
4. Tao Qin
5. Tie-Yan Liu
6. Renqian Luo

Citations in the corpus

• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (2021, background): “Approximated proxy inference such as truncated training/early stopping can accelerate the search, but is well known to introduce search bias to the inaccurate results obtained (Pham et al., 2018; Liang et al., 2019; Tan et al., 2020).”
• Neural Architecture Search with GBDT (2020, background): “Lots of works are emerged to explore different search algorithms including reinforcement learning [38, 16], evolutionary algorithm [28, 15, 19, 18], Bayesian optimization [36], performance prediction [1, 10, 14, 27] and gradient based optimization [14, 11, 13].”
• Lightspeech: Lightweight and Fast Text to Speech with Neural Architecture Search (2021, methodology): “Following [20], we adopt different kernel sizes of {1, 5, 9, 13, 17, 21, 25}.”