Evaluating the Search Phase of Neural Architecture Search
By Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Cristian Musat, Mathieu Salzmann. 2019
375 citations recorded by the source corpus.
Neural Architecture Search (NAS) aims to facilitate the design of deep networks for new tasks. Existing techniques rely on two stages: searching over the architecture space and validating the best architecture. NAS algorithms are currently compared solely based on their results on the downstream task. While intuitive, this fails to explicitly evaluate the effectiveness of their search strategies. In this paper, we propose to evaluate the NAS search phase. To this end, we compare the quality of the solutions obtained by NAS search policies with that of random architecture selection. We find that: (i) On average, the state-of-the-art NAS algorithms perform similarly to the random policy; (ii) the widely-used weight sharing strategy degrades the ranking of the NAS candidates to the point of not reflecting their true performance, thus reducing the effectiveness of the search process. We believe that our evaluation framework will be key to designing NAS strategies that consistently discover architectures superior to random ones.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Dacheng Tao
- Sergey Levine
- Luke S. Zettlemoyer
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Yi Yang
- Alan L. Yuille
- Trevor Darrell
- Jitendra Malik
In Evaluation & Benchmarks
- Colin White
- Yash Savani
- Xuanyi Dong
- Kaiming He
- Quoc V. Le
- Amos J. Storkey
- Willie Neiswanger
- Bogdan Gabrys
In Neural Architecture Search & AutoML
Citations in the corpus
- Primer: Searching for Efficient Transformers for Language Modeling (2021, background): “Our search space is different from previous search spaces (see architecture search survey by [31]), which are often heavily biased such that random search performs well (see analysis by [32, 33, 34]).”
- Weight Agnostic Neural Networks (2019, background): “It has been shown that random search can already achieve SOTA results if such priors are used [63, 88, 97].”
- Benchmarking Deep Learning Models and Automated Model Design for COVID-19 Detection with Chest CT Scans (2020, background): “In recent studies [35]–[37], the authors demonstrate that random search is a more competitive method than many others.”
- A Comprehensive Survey of Neural Architecture Search (2020, methodology): “Although RS has proven to be a strong baseline [24, 33, 104, 127], relevant research is still insufficient.”
- Semi-Supervised Neural Architecture Search (2020, background): “While greatly reducing the computational cost, the quality of the training data (architectures and their corresponding accuracy) for the controller is degraded [24], and thus these approaches suffer from accuracy decline on downstream tasks.”
- DSNAS: Direct Neural Architecture Search Without Parameter Retraining (2020, methodology): “Unfortunately, under this criterion, most existing two-stage NAS methods are reported [20, 11] failing to prove the correlation between the searching accuracy and the retraining accuracy.”
- BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, background): “Perhaps the most common type of search space for NAS is a cell-based search space [54, 34, 29, 23, 36, 51].”
- Understanding and Robustifying Differentiable Architecture Search (2019, background): “However, several researchers have also reported DARTS to not work well, in some cases even no better than random search [Li and Talwalkar, 2019, Sciuto et al., 2019].”
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2019, background): “Despite its simplicity, researchers detected some of its drawbacks, such as instability (Li & Talwalkar, 2019; Sciuto et al., 2019), which led to a few improved approaches beyond DARTS (Cai et al.”
- FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (2019, background): “A recent work [20] evaluates the search phase based on Kendall Tau [10], which measures the ranking relation between models during the search phase and fully trained ones.”
- A Survey on Neural Architecture Search (2019, methodology): “h can be considered as a special way of optimizing hyperparameters, most of the related work is disregarded. In particular, random search which has proven to be an extremely strong baseline. In fact, Sciuto et al. (2019) show that random search nds better RNN cells than any other optimizer. Li and T”
- Evolutionary-Neural Hybrid Agents for Architecture Search (2018, methodology): “These methods can only be applied to a narrow subset of all architecture search problems [24, 36].”
- Neural Architecture Search Using Deep Neural Networks and Monte Carlo Tree Search (2018, neutral): “We choose Random Search (RS) (Sciuto et al. 2019) and Regularized Evolution (RE) (Real et al.”
- Single Path One-Shot Neural Architecture Search with Uniform Sampling (2019, background): “Some papers [27, 25, 19] even show that several the state-of-the-art NAS methods perform similarly to the random search.”
- Exploring Randomly Wired Neural Networks for Image Recognition (2019, methodology): “works tend to have certain properties. The generator design underlies the prior and thus should not be overlooked. Our work explores a direction orthogonal to concurrent work on random search for NAS [23,41]. These studies show that random search is competitive in “the NAS search space” [55,56], i.e”