Random Search and Reproducibility for Neural Architecture Search

By Liam Li, Ameet Talwalkar. 2019

807 citations recorded by the source corpus.

Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. In this work, in order to help ground the empirical results in this field, we propose new NAS baselines that build off the following observations: (i) NAS is a specialized hyperparameter optimization problem; and (ii) random search is a competitive baseline for hyperparameter optimization. Leveraging these observations, we evaluate both random search with early-stopping and a novel random search with weight-sharing algorithm on two standard NAS benchmarks---PTB and CIFAR-10. Our results show that random search with early-stopping is a competitive NAS baseline, e.g., it performs at least as well as ENAS, a leading NAS method, on both benchmarks. Additionally, random search with weight-sharing outperforms random search with early-stopping, achieving a state-of-the-art NAS result on PTB and a highly competitive result on CIFAR-10. Finally, we explore the existing reproducibility issues of published NAS results. We note the lack of source material needed to exactly reproduce these results, and further discuss the robustness of published results given the various sources of variability in NAS experimental setups. Relatedly, we provide all information (code, random seeds, documentation) needed to exactly reproduce our results, and report our random search with weight-sharing results for each benchmark on multiple runs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Quoc V. Le
  2. Mohit Bansal
  3. Yoshua Bengio
  4. Percy Liang
  5. Dacheng Tao
  6. Jianfeng Gao
  7. Luke S. Zettlemoyer
  8. Sergey Levine

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Yi Yang
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Trevor Darrell
  7. Alan L. Yuille
  8. Jitendra Malik

In Evaluation & Benchmarks

  1. Colin White
  2. Yash Savani
  3. Frank Hutter
  4. Sam Nolen
  5. Xuanyi Dong
  6. Willie Neiswanger
  7. Quoc V. Le
  8. Yi Yang

In Neural Architecture Search & AutoML

  1. Frank Hutter
  2. Colin White
  3. Yash Savani
  4. Sam Nolen
  5. Xuanyi Dong
  6. Willie Neiswanger
  7. Quoc V. Le
  8. Arber Zela

Citations in the corpus