NAS-Bench-101: Towards Reproducible Neural Architecture Search
By Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin P. Murphy, Frank Hutter. 2019
821 citations recorded by the source corpus.
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build NAS-Bench-101, we carefully constructed a compact, yet expressive, search space, exploiting graph isomorphisms to identify 423k unique convolutional architectures. We trained and evaluated all of these architectures multiple times on CIFAR-10 and compiled the results into a large dataset of over 5 million trained models. This allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset. We demonstrate its utility by analyzing the dataset as a whole and by benchmarking a range of architecture optimization algorithms.
Subfields: Artificial Intelligence, Information Systems, Materials Chemistry, Evaluation & Benchmarks, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Dacheng Tao
- Sergey Levine
- Percy Liang
- Luke S. Zettlemoyer
In Information Systems
In Materials Chemistry
- Yash Savani
- Colin White
- Sam Nolen
- Frank Hutter
- Willie Neiswanger
- Xuanyi Dong
- Marius Thomas Lindauer
- Xia Hu
In Evaluation & Benchmarks
- Frank Hutter
- Colin White
- Yash Savani
- Sam Nolen
- Marius Thomas Lindauer
- Xuanyi Dong
- Yi Yang
- Quoc V. Le
In Neural Architecture Search & AutoML
- Colin White
- Yash Savani
- Frank Hutter
- Sam Nolen
- Willie Neiswanger
- Marius Thomas Lindauer
- Xuanyi Dong
- Quoc V. Le
Citations in the corpus
- Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “…and used them in various settings: analyzing population-level trends (Radosavovic et al., 2019; 2020), benchmarking neural architecture search (Ying et al., 2019), training hypernetworks (Knyazev et al., 2021), predicting model properties (Sch¨urholt et al., 2021) and dataset distillation…”
- AutoML: A Survey of the State-of-the-Art (2019, background): “NASBench [139] goes a step further and is a pioneering work to alleviate the problem of non-reproducibility.”
- NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, methodology): “NAS-Bench-101 [45] comprises 423k unique convolutional architectures trained on CIFAR-10 image dataset.”
- AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, neutral): “For example, all models in [55, 47, 53] are trained with the same optimizer, learning rate, and weight decay.”
- A Comprehensive Survey of Neural Architecture Search (2020, background): “In the work of SNAS, DARTS and many other NAS [75–78], the feasible paths of the searched network architecture depend on each other and are closely coupled during the search phase.”
- AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, methodology): “stated, we use the regularized evolution search method because of its simplicity and recent success on architecture search benchmarks (Real et al., 2019; Ying et al., 2019; So et al., 2019).”
- Semi-Supervised Neural Architecture Search (2020, methodology): “Dataset NASBench-101 [37] designs a cell-based search space following the common practice [42, 17, 15].”
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search (2020, methodology): “To the best of our knowledge, NAS-Bench-101 (Ying et al., 2019) is the only existing large-scale architecture dataset.”
- NAS evaluation is frustratingly hard (2019, background): “To this end, NAS-Bench-101 (Ying et al., 2019), a dataset mapping architectures to their accuracy, can be extremely useful, as it allows the quality of search strategies to be assessed in isolation from other NAS components (e.g. search space, training protocol) in a quick and reproducible fashion.”
- BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “The NASBench101 dataset (Ying et al. 2019) consists of over 423,000 neural architectures from a cell-based search space, and each archi- tecture comes with precomputed validation and test accuracies on CIFAR-10.”
- Towards modular and programmable architecture search (2019, methodology): “We refer to the search spaces we consider as Nasbench [27], Nasnet [28], Flat [15], and Genetic [26].”
- Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization (2019, methodology): “Orthogonally to this work, we performed an exhaustive search for a large neural architecture search problem and compiled all architecture and performance pairs into a neural architecture search benchmark (Ying et al., 2019).”
- Evaluating the Search Phase of Neural Architecture Search (2019, methodology): “Ying et al. (2019) introduced a dataset that contains the ground-truth performance of CNN cells, and Wang et al. (2019) evaluated some traditional search algorithms on it.”
- Evolutionary-Neural Hybrid Agents for Architecture Search (2018, positive): “We perform the same comparison using NASBench [47], where the agents search for network architectures for CIFAR-10 classification task in a confined search space, which consists of approximately 423k unique architectures.”
- Neural Architecture Search Using Deep Neural Networks and Monte Carlo Tree Search (2018, positive): “Searching on NAS dataset: To further examine the sample efficiency, we evaluate AlphaX on the recent NAS dataset, NASBench-101 (Ying et al. 2019).”
- Meta-Learning in Neural Networks: A Survey (2020, background): “support reproducible and accessible research, the NASbenches [245] provide pre-computed performance measures for a large number of network architectures.”
- Best Practices for Scientific Research on Neural Architecture Search (2019, methodology): “Especially in such cases, we advocate running extensive evaluations on tabular benchmarks, such as NAS-Bench-101 (Ying et al., 2019) and the NAS-HPO benchmarks (Klein & Hutter, 2019), or on surrogate benchmarks as proposed by Eggensperger et al. (2015; 2018) and used by Falkner et al. (2018).”
- Automated Machine Learning: From Principles to Practices (2018, methodology): “These methods constructed CNN [86, 158, 174] , RNN [74], transformers [125] with the repeated cells. figure for organization in applications 3.2.3 2/3D mesh.”