Neural Architecture Search without Training

By J. Mellor, Jack Turner, Amos J. Storkey, Elliot J. Crowley. 2020

496 citations recorded by the source corpus.

The time and effort involved in hand-designing deep neural networks is immense. This has prompted the development of Neural Architecture Search (NAS) techniques to automate this design. However, NAS algorithms tend to be extremely slow and expensive; they need to train vast numbers of candidate networks to inform the search process. This could be remedied if we could infer a network's trained accuracy from its initial state. In this work, we examine how the linear maps induced by data points correlate for untrained network architectures in the NAS-Bench-201 search space, and motivate how this can be used to give a measure of modelling flexibility which is highly indicative of a network's trained performance. We incorporate this measure into a simple algorithm that allows us to search for powerful networks without any training in a matter of seconds on a single GPU. Code to reproduce our experiments is available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “Furthermore, when training is decoupled from NAS, sub-networks tailoring to specialized system hardware can be selected without additional training (Stamoulis et al., 2019; Cai et al., 2020a; Chen et al., 2021; Mellor et al., 2021).”
Auto-scaling Vision Transformers without Training (2022, methodology): “We note a recent surge of training-free neural architecture search methods for ReLU-based CNNs, leveraging local linear maps (Mellor et al., 2020), gradient sensitivity (Abdelfattah et al., 2021), number of linear regions (Chen et al., 2021e;f), or network topology (Bhardwaj et al., 2021).”
Zero-Cost Proxies for Lightweight NAS (2021, methodology): “3 JACOBIAN COVARIANCE This metric was purpose-designed to score neural networks in the context of NAS – we refer the reader to the original paper for detailed reasoning and derivation of the metric which we call jacob cov (Mellor et al., 2020).”
Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks (2022, background): “However, recent studies [34] showed how to improve the search algorithm, making it faster to discover new architectures.”
Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (2021, background): “Although some heavy networks can theoretically represent complex functions, they not necessarily can be effectively trained by gradient descent.”
pTNAS: Progressive Neural Architecture Search for Tabular Data (2024, unknown): “Likewise, NASWOT (Mellor et al., 2021) characterizes expressivity by measuring the distance between the vectors of activation patterns for any two samples within a batch.”