NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size

By Xuanyi Dong, Lu Liu, Katarzyna Musial, Bogdan Gabrys. 2020

213 citations recorded by the source corpus.

Neural architecture search (NAS) has attracted a lot of attention and has been illustrated to bring tangible benefits in a large number of applications in the past few years. Architecture topology and architecture size have been regarded as two of the most important aspects for the performance of deep learning models and the community has spawned lots of searching algorithms for both of those aspects of the neural architectures. However, the performance gain from these searching algorithms is achieved under different search spaces and training setups. This makes the overall performance of the algorithms incomparable and the improvement from a sub-module of the searching model unclear. In this paper, we propose NATS-Bench, a unified benchmark on searching for both topology and size, for (almost) any up-to-date NAS algorithm. NATS-Bench includes the search space of 15,625 neural cell candidates for architecture topology and 32,768 for architecture size on three datasets. We analyze the validity of our benchmark in terms of various criteria and performance comparison of all candidates in the search space. We also show the versatility of NATS-Bench by benchmarking 13 recent state-of-the-art NAS algorithms on it. All logs and diagnostic information trained using the same setup for each candidate are provided. This facilitates a much larger community of researchers to focus on developing better NAS algorithms in a more comparable and computationally effective environment. All codes are publicly available at: https://xuanyidong.com/assets/projects/NATS-Bench.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Materials Chemistry

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Materials Chemistry

Citations in the corpus

A Comprehensive Survey of Neural Architecture Search (2020, methodology): “Some works have also been devoted to providing a unified evaluation platform for popular NAS methods [79, 128].”
Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (2021, background): “These two questions are entangled and highly non-trivial, since the search spaces are of extremely high dimension, and the generalization ability of architectures cannot be easily inferred (Dong & Yang, 2020; Dong et al., 2020).”
Few-Shot Learning With a Strong Teacher (2021, background): “As shown in several recent works like [113], [9], [23], the backbone architectures of the feature extractor can largely affect the resulting classification or FSL accuracy.”
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, methodology): “Cell-based search spaces, ﬁrst proposed in [93], are generally used in previous NAS meth-ods [42, 54, 43, 52] and benchmarks [74, 19, 17].”
Neural Architecture Search without Training (2020, methodology): “2 The plots for NATS-Bench SSS can be found in Figure 9 (Appendix B).”