NAS evaluation is frustratingly hard

By Antoine Yang, Pedro M. Esperança, Fabio Maria Carlucci. 2019

181 citations recorded by the source corpus.

Neural Architecture Search (NAS) is an exciting new field which promises to be as much as a game-changer as Convolutional Neural Networks were in 2012. Despite many great works leading to substantial improvements on a variety of tasks, comparison between different methods is still very much an open issue. While most algorithms are tested on the same datasets, there is no shared experimental protocol followed by all. As such, and due to the under-use of ablation studies, there is a lack of clarity regarding why certain methods are more effective than others. Our first contribution is a benchmark of 8 NAS methods on 5 datasets. To overcome the hurdle of comparing methods with different search spaces, we propose using a method’s relative improvement over the randomly sampled average architecture, which effectively removes advantages arising from expertly engineered search spaces or training protocols. Surprisingly, we find that many NAS techniques struggle to significantly beat the average architecture baseline. We perform further experiments with the commonly used DARTS search space in order to understand the contribution of each component in the NAS pipeline. These experiments highlight that: (i) the use of tricks in the evaluation protocol has a predominant impact on the reported performance of architectures; (ii) the cell-based search space has a very narrow accuracy range, such that the seed has a considerable impact on architecture rankings; (iii) the hand-designed macrostructure (cells) is more important than the searched micro-structure (operations); and (iv) the depth-gap is a real phenomenon, evidenced by the change in rankings between 8 and 20 cell architectures. To conclude, we suggest best practices, that we hope will prove useful for the community and help mitigate current NAS pitfalls, e.g. difficulties in reproducibility and comparison of search methods. We provide the code used for our experiments at link-to-come.

Subfields: Biomedical Engineering, Molecular Biology, Radiology, Nuclear Medicine and Imaging, Evaluation & Benchmarks, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Molecular Biology

In Radiology, Nuclear Medicine and Imaging

In Evaluation & Benchmarks

In Neural Architecture Search & AutoML

Citations in the corpus

NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, background): “[44] notice: most researchers do not strictly follow the experimental protocol or do not make enough ablation studies, leaving obscure reasons for the effectiveness of various methods.”
A Comprehensive Survey of Neural Architecture Search (2020, methodology): “Although RS has proven to be a strong baseline [24, 33, 104, 127], relevant research is still insufficient.”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “Indeed, dominant aspects of performance are often left out (Yang et al., 2020).”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “The NASBench-201 dataset [71] consists of 15625 neural architectures with precomputed validation and test accuracies for 200 epochs on CIFAR-10, CIFAR-100, and ImageNet-16-120.”
Single Path One-Shot Neural Architecture Search with Uniform Sampling (2019, background): “Some papers [27, 25, 19] even show that several the state-of-the-art NAS methods perform similarly to the random search.”
Best Practices for Scientific Research on Neural Architecture Search (2019, methodology): “tfalls we mention and try to avoid. We note that there are several recent papers that cast doubts on much of the work in the eld of NAS (Sciuto et al., 2019; Li and Talwalkar, 2019; Xie et al., 2019; Yang et al., 2020), and these have led to serious scepticism of outsiders concerning NAS. In this pa”
Neural Ensemble Search for Performant and Calibrated Predictions (2020, background): “1 NES with Random Search In NAS, random search (RS) is a competitive baseline on carefully designed architecture search spaces [35, 57, 58].”
Revisiting the Train Loss: an Efficient Performance Estimator for Neural Architecture Search (2020, background): “However, the weight sharing ranking among architectures often correlates very poorly with the true performance ranking [16, 17, 27], meaning architectures chosen by one-shot NAS are likely to be sub-optimal when evaluated independently [27].”
Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS (2020, methodology): “We compare BONAS with the following state-of-the-art sample-based NAS baselines: (i) Random search [37], which explores the search space randomly without exploitation; (ii) Regularized evolution [25], which uses a heuristic evolution process for exploitation; (iii) NASBOT [11], which uses BO with a…”
One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, background): “[58] conducted extensive experiments that demonstrated that the current one-shot NAS techniques struggle to outperform naive baselines.”
Local Search is State of the Art for Neural Architecture Search Benchmarks (2020, methodology): “Not only is this a vicious cycle (will we start using AutoML algorithms to tune AutoML algorithms?), but the runtime for any hyper-hyperparameter tuning on a new dataset must be added to the total runtime of the NAS algorithm [19, 38].”
Block-Wisely Supervised Neural Architecture Search With Knowledge Distillation (2019, background): “Typically, [26] and [35] even suggest that many existing solutions to NAS are no better than or struggle to outperform random architecture selection.”
DARTS-: Robustly Stepping out of Performance Collapse Without Indicators (2020, unknown): “To verify the robustness, it is required to search several times to report the average performance of derived models (Yu et al., 2020; Antoine et al., 2020).”
Local Search is State of the Art for NAS Benchmarks (2020, methodology): “Not only is this a vicious cycle (will we start using AutoML algorithms to tune AutoML algorithms?), but the runtime for any hyper-hyperparameter tuning on a new dataset must be added to the total runtime of the NAS algorithm (Lindauer and Hutter, 2019; Yang et al., 2020).”
Neural Architecture Search with GBDT (2020, background): “Search Space Search Search space plays an essential role in the search phase [34, 31].”
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, background): “As pointed out in the literature [58, 73, 80], weight-sharing methods suffer from low architecture rating accuracy.”