Random Search and Reproducibility for Neural Architecture Search
By Liam Li, Ameet Talwalkar. 2019
807 citations recorded by the source corpus.
Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. In this work, in order to help ground the empirical results in this field, we propose new NAS baselines that build off the following observations: (i) NAS is a specialized hyperparameter optimization problem; and (ii) random search is a competitive baseline for hyperparameter optimization. Leveraging these observations, we evaluate both random search with early-stopping and a novel random search with weight-sharing algorithm on two standard NAS benchmarks---PTB and CIFAR-10. Our results show that random search with early-stopping is a competitive NAS baseline, e.g., it performs at least as well as ENAS, a leading NAS method, on both benchmarks. Additionally, random search with weight-sharing outperforms random search with early-stopping, achieving a state-of-the-art NAS result on PTB and a highly competitive result on CIFAR-10. Finally, we explore the existing reproducibility issues of published NAS results. We note the lack of source material needed to exactly reproduce these results, and further discuss the robustness of published results given the various sources of variability in NAS experimental setups. Relatedly, we provide all information (code, random seeds, documentation) needed to exactly reproduce our results, and report our random search with weight-sharing results for each benchmark on multiple runs.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Mohit Bansal
- Yoshua Bengio
- Percy Liang
- Dacheng Tao
- Jianfeng Gao
- Luke S. Zettlemoyer
- Sergey Levine
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Xiaogang Wang
- Ming-Hsuan Yang
- Trevor Darrell
- Alan L. Yuille
- Jitendra Malik
In Evaluation & Benchmarks
In Neural Architecture Search & AutoML
Citations in the corpus
- Primer: Searching for Efficient Transformers for Language Modeling (2021, background): “Our search space is different from previous search spaces (see architecture search survey by [31]), which are often heavily biased such that random search performs well (see analysis by [32, 33, 34]).”
- Show Your Work: Improved Reporting of Experimental Results (2019, background): “Moreover, recent investigations into “state-ofthe-art” claims have found competing methods to only be comparable, without clear superiority, even against baselines (Reimers and Gurevych, 2017; Lucic et al., 2018; Li and Talwalkar, 2019); this has exposed the need for reporting more than a single poi”
- AutoML: A Survey of the State-of-the-Art (2019, methodology): “The authors of [145] further demonstrate that random search with weight-sharing can outperform a series of powerful methods such as ENAS and DARTS, achieving SOTA results on PTB and a highly competitive result”
- Weight Agnostic Neural Networks (2019, background): “It has been shown that random search can already achieve SOTA results if such priors are used [63, 88, 97].”
- NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, background): “For instance, Li and Talwalkar [26] have been unable to exactly reproduce state-of-the-art (SOTA) NAS methods due to the lack of source code and data.”
- A Comprehensive Survey of Neural Architecture Search (2020, methodology): “Although RS has proven to be a strong baseline [24, 33, 104, 127], relevant research is still insufficient.”
- GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (2020, methodology): “The sampling can be uniform sampling [11, 16] or multi-path sampling with fairness [4].”
- AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “For example, comparisons on the same space found that advanced techniques are often only marginally superior to simple random search (RS) (Li & Talwalkar, 2019; Elsken et al., 2019b; Negrinho et al., 2019).”
- DSNAS: Direct Neural Architecture Search Without Parameter Retraining (2020, methodology): “On the other side of the spectrum, one-shot methods such as random search [11] and SPOS [7] can be extended to single-stage training.”
- On Robustness of Neural Architecture Search Under Label Noise (2020, background): “The observation has also been pointed out in Li and Talwalkar (2019).”
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search (2020, methodology): “Specifically, we evaluate some typical NAS algorithms: (I) Random Search algorithms, e.g., random search (RS) (Bergstra & Bengio, 2012), random search with parameter sharing (RSPS) (Li & Talwalkar, 2019).”
- NAS evaluation is frustratingly hard (2019, methodology): “Recent works have warned that this procedure might negatively affect ranking; similarly, the role of the seed has been previously recognized as as a fundamental element in reproducibility (Li & Talwalkar, 2019; Sciuto et al., 2019).”
- BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “We compare BANANAS to various popular NAS algorithms: random search (Li and Talwalkar 2019), regularized evolution (Real et al.”
- Understanding and Robustifying Differentiable Architecture Search (2019, methodology): “Table 1 confirms the very poor performance standard DARTS yields for all of these settings also compared to Random Search with weight sharing (RS-ws) [Li and Talwalkar, 2019].”
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2019, background): “Despite its simplicity, researchers detected some of its drawbacks, such as instability (Li & Talwalkar, 2019; Sciuto et al., 2019), which led to a few improved approaches beyond DARTS (Cai et al., 2019; Chen et al., 2019; Mei et al., 2020).”
- FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (2019, neutral): “approach doesn’t suffer [22].”