NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
By Xuanyi Dong, Yi Yang. 2020
863 citations recorded by the source corpus.
Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years. It could be time to take a step back and analyze the good and bad aspects in the field of NAS. A variety of algorithms search architectures under different search space. These searched architectures are trained using different setups, e.g., hyper-parameters, data augmentation, regularization. This raises a comparability problem when comparing the performance of various NAS algorithms. NAS-Bench-101 has shown success to alleviate this problem. In this work, we propose an extension to NAS-Bench-101: NAS-Bench-201 with a different search space, results on multiple datasets, and more diagnostic information. NAS-Bench-201 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms. The design of our search space is inspired by the one used in the most popular cell-based searching algorithms, where a cell is represented as a directed acyclic graph. Each edge here is associated with an operation selected from a predefined operation set. For it to be applicable for all NAS algorithms, the search space defined in NAS-Bench-201 includes all possible architectures generated by 4 nodes and 5 associated operation options, which results in 15,625 neural cell candidates in total. The training log using the same setup and the performance for each architecture candidate are provided for three datasets. This allows researchers to avoid unnecessary repetitive training for selected architecture and focus solely on the search algorithm itself. The training time saved for every architecture also largely improves the efficiency of most NAS algorithms and presents a more computational cost friendly NAS community for a broader range of researchers. We provide additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms. In further support of the proposed NAS-Bench-102, we have analyzed it from many aspects and benchmarked 10 recent NAS algorithms, which verify its applicability.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Dacheng Tao
- Xiaodan Liang
- Jianfeng Gao
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Xiaogang Wang
- Trevor Darrell
- Alan L. Yuille
- Xiaodan Liang
- Bernt Schiele
In Evaluation & Benchmarks
- Xuanyi Dong
- Colin White
- Yash Savani
- Marius Thomas Lindauer
- Mingxing Tan
- Sam Nolen
- Amos J. Storkey
- Bogdan Gabrys
In Neural Architecture Search & AutoML
- Colin White
- Yash Savani
- Sam Nolen
- Xiaojun Chang
- Willie Neiswanger
- Bogdan Gabrys
- Marius Thomas Lindauer
- Bo Zhang
Citations in the corpus
- NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, methodology): “NAS-Bench-201 [11] by Dong and Yang proposed a framework and ran ten popular NAS algorithms within it.”
- AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, background): “This strategy has an over-fitting problem, which is also found in NAS [11, 52, 12, 50].”
- BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “In the full version of the paper, we show a similar result for NASBench-201 (Dong and Yang 2020): the full path encoding length of ∑3 i=0 5 i = 156 can be truncated to just 30, without a loss of performance.”
- FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (2019, methodology): “Our analysis and experiments are conducted in a widely used search space as in [4, 48, 16, 41], as well as a cell-based search space from a common benchmark NAS-Bench-201 [14].”
- Meta-Learning in Neural Networks: A Survey (2020, background): “To support reproducible and accessible research, the recently released NASbenches [241], [242] provides pre-computed performance measures for a large number of network architectures.”
- Best Practices for Scientific Research on Neural Architecture Search (2019, methodology): “peline of DARTS (Liu et al., 2019b), evaluated on CIFAR-10 (with standard training/test split).6 Example 5 NAS-Bench-101 (Ying et al., 2019), NAS-Bench-1Shot1 (Zela et al., 2020b), and NAS-Bench-201 (Dong and Yang, 2020) are tabular NAS benchmarks that, on top of a publicly available search space, t”
- A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, background): “Recently, a large volume of work [112], [113], [114] has shown that performance differences still exist between one-shot NAS and traditional NAS.”
- Local Search is State of the Art for NAS Benchmarks (2020, methodology): “, 2020), spurring the release of two NAS benchmark datasets (Ying et al., 2019; Dong and Yang, 2020).”
- Neural Architecture Search without Training (2020, methodology): “We compute K H for a random subset of NAS-Bench-201 (Dong & Yang, 2020) and NDS-DARTS (Radosavovic et al., 2019) networks at initialisation for a mini-batch of CIFAR-10 images.”
- iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, methodology): “We empirically find that T = 4 is enough to achieve competitive results on NAS-Bench-201.”
- Adversarially Robust Neural Architectures (2020, neutral): “Fortheevaluation,we setthetotalperturbationsizetobe : We further verify the searching effectiveness of RACL on a popular benchmark NAS-bench-201 [55], where all the possible architecture candidates within a predefined search space and their corresponding retrained weights are available for a fair…”
- Local Search is State of the Art for Neural Architecture Search Benchmarks (2020, background): “NASBench-201 [8].”
- Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search (2020, methodology): “Since NB101 is somewhat harder than NB201, we allocate 500 queries for NB101 and 200 queries for NB201 including 10% of random selection at the beginning of BO.”
- Revisiting the Train Loss: an Efficient Performance Estimator for Neural Architecture Search (2020, methodology): “We adopt Spearman’s rank correlation following [35, 28].”
- One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, methodology): “Although we demonstrate the theoretical benefits of the NSAS loss function in relieving catastrophic forgetting TABLE 3 Results of One-Shot NAS Baselines on NAS-Bench-201 Method CIFAR-10 CIFAR-100 ImageNet-16-120 Valid(”
- Zero-Cost Proxies for Lightweight NAS (2021, methodology): “The only difference to the standard NAS-Bench-201 training pipeline (Dong & Yang, 2020) is our use of fewer epochs for the learning rate annealing schedule – we anneal the learning rate to zero over 40 epochs instead of 200.”