Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective

By Wuyang Chen, Xinyu Gong, Zhangyang Wang. 2021

298 citations recorded by the source corpus.

Neural Architecture Search (NAS) has been explosively studied to automate the discovery of top-performer neural networks. Current works require heavy training of supernet or intensive architecture evaluations, thus suffering from heavy resource consumption and often incurring search bias due to truncated training or approximations. Can we select the best neural architectures without involving any training and eliminate a drastic portion of the search cost? We provide an affirmative answer, by proposing a novel framework called training-free neural architecture search (TE-NAS). TE-NAS ranks architectures by analyzing the spectrum of the neural tangent kernel (NTK) and the number of linear regions in the input space. Both are motivated by recent theory advances in deep networks and can be computed without any training and any label. We show that: (1) these two measurements imply the trainability and expressivity of a neural network; (2) they strongly correlate with the network's test accuracy. Further on, we design a pruning-based NAS mechanism to achieve a more flexible and superior trade-off between the trainability and expressivity during the search. In NAS-Bench-201 and DARTS search spaces, TE-NAS completes high-quality search but only costs 0.5 and 4 GPU hours with one 1080Ti on CIFAR-10 and ImageNet, respectively. We hope our work inspires more attempts in bridging the theoretical findings of deep networks and practical impacts in real NAS applications. Code is available at: https://github.com/VITA-Group/TENAS.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Neural Architecture Search & AutoML

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “Furthermore, when training is decoupled from NAS, sub-networks tailoring to specialized system hardware can be selected without additional training (Stamoulis et al., 2019; Cai et al., 2020a; Chen et al., 2021; Mellor et al., 2021).”
Dataset Distillation with Infinitely Wide Convolutional Networks (2021, background): “…[Adlam and Pennington, 2020], training dynamics [Lewkowycz et al., 2020, Lewkowycz and Gur-Ari, 2020], uncertainty [Adlam et al., 2021], and demonstrated their effectiveness for smaller datasets [Arora et al., 2020] and neural architecture search [Park et al., 2020, Chen et al., 2021].”
When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations (2021, background): “If κ diverges then the network will become untrainable (Xiao et al., 2020; Chen et al., 2021a).”
Auto-scaling Vision Transformers without Training (2022, methodology): “In addition to L E , we also include the NTK condition number κ Θ = λ max λ min to indicate the trainability of ViTs (Chen et al., 2021e; Xiao et al., 2019; Yang, 2020; Hron et al., 2020). λ max and λ min are the largest and smallest eigenvalue of NTK matrix Θ .”
Neural Architecture Search without Training (2020, background): “Chen et al. (2021) propose two further heuristics—one based on the condition number of the neural tangent kernel (Jacot et al., 2018) at initialisation and the other based on the number of unique linear regions that partition training data at initialisation—with a proposed strategy to combine these…”
pTNAS: Progressive Neural Architecture Search for Tabular Data (2024, unknown): “The application of the Neural Tangent Kernel (NTK) as a measure of trainability has recently been explored, given a batch of data X B , NTK (Jacot et al., 2018; Arora et al., 2019; Allen-Zhu et al., 2019) characterizes the complexities of the training dynamics at initialization, which is defined as ”