Neural Predictor for Neural Architecture Search

By Wujie Wen, Hanxiao Liu, Hai Helen Li, Yiran Chen, Gabriel Bender, Pieter-Jan Kindermans. 2019

238 citations recorded by the source corpus.

Neural Architecture Search methods are effective but often use complex algorithms to come up with the best architecture. We propose an approach with three basic steps that is conceptually much simpler. First we train N random architectures to generate N (architecture, validation accuracy) pairs and use them to train a regression model that predicts accuracy based on the architecture. Next, we use this regression model to predict the validation accuracies of a large number of random architectures. Finally, we train the top-K predicted architectures and deploy the model with the best validation result. While this approach seems simple, it is more than 20 times as sample efficient as Regularized Evolution on the NASBench-101 benchmark and can compete on ImageNet with more complex approaches based on weight sharing, such as ProxylessNAS.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Materials Chemistry, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Materials Chemistry

In Neural Architecture Search & AutoML

Citations in the corpus

FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, methodology): “Utilizing the Huber loss prevents the model from being dominated by outliers, as observed in [37].”
Semi-Supervised Neural Architecture Search (2020, background): “In the scenario of NAS, unlabeled architectures can be obtained through random generation, mutation [20], or simply going through the whole search space [32], which incur nearly zero additional cost.”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, background): “2018), GCN Predictor (Wen et al. 2019), and DNGO (Snoek et al.”
NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search (2020, methodology): “Shi et al. (2019) and Wen et al. (2019) trained a Graph Convolutional Network (GCN) on a subset of NAS-Bench-101 (Ying et al., 2019) showing its effectiveness in predicting the performance of unseen architectures.”
Local Search is State of the Art for NAS Benchmarks (2020, methodology): “Many of the recently proposed state-of-the-art algorithms are becoming increasingly more complex, most of which use neural networks as subroutines (Wen et al., 2019; White et al., 2019).”
Lightspeech: Lightweight and Fast Text to Speech with Neural Architecture Search (2021, methodology): “Thirdly, among many well-performing NAS algorithms, we adopt accuracy prediction based NAS [17, 18] which is straightforward and efﬁcient.”
Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (2021, background): “Graph neural network (GNN) is a popular choice as the predictor model (Wen et al., 2019; Chen et al., 2020b).”
Local Search is State of the Art for Neural Architecture Search Benchmarks (2020, background): “Most of the recently proposed state-of-the-art algorithms are becoming increasingly more complex, many of which use neural networks as subroutines [35, 36].”
NPENAS: Neural Predictor Guided Evolution for Neural Architecture Search (2022, methodology): “[12] adopted a complicated cascade neural predictor to rank all the architectures in search space, select the top-k architectures”
Parameter Prediction for Unseen Deep Architectures (2021, methodology): “We compare to the Neural Predictor (NeuPred) [80].”
Neural Architecture Search with GBDT (2020, methodology): “From the statistics of the dataset [32, 27], the best test accuracy is 94 .”