NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search

By Arber Zela, Julien N. Siems, Frank Hutter. 2020

158 citations recorded by the source corpus.

One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice. Nevertheless, there is still a lack of understanding on how these weight-sharing algorithms exactly work due to the many factors controlling the dynamics of the process. In order to allow a scientific study of these components, we introduce a general framework for one-shot NAS that can be instantiated to many recently-introduced variants and introduce a general benchmarking framework that draws on the recent large-scale tabular benchmark NAS-Bench-101 for cheap anytime evaluations of one-shot NAS methods. To showcase the framework, we compare several state-of-the-art one-shot NAS methods, examine how sensitive they are to their hyperparameters and how they can be improved by tuning their hyperparameters, and compare their performance to that of blackbox optimizers for NAS-Bench-101.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, background): “[47] built a one-shot NAS framework on the top of NAS-Bench-101 that can reuse the underlying computations.”
A Comprehensive Survey of Neural Architecture Search (2020, methodology): “Although NAS-related research has been highly abundant, it is still difficult to compare and reproduce NAS methods [79, 128, 129, 156].”
NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search (2020, methodology): “101 without any modification is only applicable to selected algorithms (Yu et al., 2020; Zela et al., 2020).”
Automated Machine Learning: From Principles to Practices (2018, background): “Additionally, the choice of parameters for updating, both in proportion and in the number of layers, affects fine-tuning performance.”
Propagation Model Search for Graph Neural Networks (2020, methodology): “Note that ENAS (Pham et al. 2018) can also avoid computing redundant paths when α is updated by policy gradient, however, it has been proven to be less effective than gradient-based methods due to the instability of reinforcement learning process (Zela, Siems, and Hutter 2020).”
iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, methodology): “Different from the majority of existing works that attributes the failure of DARTS to its supernet optimization (Zela et al., 2020a; Benyahia et al., 2019), or the ﬁnal discretization with argmax (Chen & Hsieh, 2020; Wang et al., 2021b), this paper revisits DARTS from the perspective of the…”
Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? (2020, background): “identity, max-pooling) which are often preferred early on in the search, resulting in lower final accuracies [18, 19, 20, 21].”
Supplementary material for DEHB: Evolutionary Hyperband for Scalable, Robust and Efﬁcient Hyperparameter Optimization (2021, unknown): “NAS-Bench-1shot1 was introduced by [Zela et al. , 2020], as a benchmark derived from the large space of architectures offered by NAS-Bench-101.”
Neural Architecture Search without Training (2020, background): “This has led to the creation of several benchmarks (Ying et al., 2019; Zela et al., 2020; Dong & Yang, 2020; Dong et al., 2021) that consist of tractable NAS search spaces, and metadata for the training of networks within that search space.”
NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search (2020, methodology): “One result of this limitation is that One-Shot NAS methods can only be applied to subspaces of NAS-Bench-101, given in NAS-Bench-1Shot1 (Zela et al., 2020b).”
Revisiting the Train Loss: an Efficient Performance Estimator for Neural Architecture Search (2020, methodology): “However, the weight sharing ranking among architectures often correlates very poorly with the true performance ranking [16, 17, 27], meaning architectures chosen by one-shot NAS are likely to be sub-optimal when evaluated independently [27].”
NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size (2020, methodology): “We bring some results from [18] and [20] to provide some preliminary evidence of generalization.”
SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization (2021, background): “…et al., 2018), Dragonﬂy (Kandasamy et al., 2020) and BOHB (Falkner et al., 2018) on the surrogate benchmark for HPO on DNNs on the letter dataset (Falkner et al., 2018), a joint HPO+NAS benchmark (Klein and Hutter, 2019) on the Naval-Propulsion dataset and a pure NAS benchmark (Zela et al., 2020).”
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, background): “As pointed out in the literature [57, 72, 79], weight-sharing methods suffer from low architecture rating accuracy.”
One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting (2020, background): “[27], [60], [62] have tried to address this problem by building benchmark datasets for NAS.”