Searching for a Robust Neural Architecture in Four GPU Hours
By Xuanyi Dong, Yezhou Yang. 2019
719 citations recorded by the source corpus.
Conventional neural architecture search (NAS) approaches are usually based on reinforcement learning or evolutionary strategy, which take more than 1000 GPU hours to find a good model on CIFAR-10. We propose an efficient NAS approach, which learns the searching approach by gradient descent. Our approach represents the search space as a directed acyclic graph (DAG). This DAG contains thousands of sub-graphs, each of which indicates a kind of neural architecture. To avoid traversing all the possibilities of the sub-graphs, we develop a differentiable sampler over the DAG. This sampler is learnable and optimized by the validation loss after training the sampled architecture. In this way, our approach can be trained in an end-to-end fashion by gradient descent, named Gradient-based search using Differentiable Architecture Sampler (GDAS). In experiments, we can finish one searching procedure in four GPU hours on CIFAR-10, and the discovered model obtains a test error of 2.82% with only 2.5M parameters, which is on par with the state-of-the-art.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Yi Yang
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Trevor Darrell
- Andrea Vedaldi
- Andrew Zisserman
In Neural Architecture Search & AutoML
Citations in the corpus
- MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation (2020, methodology): “Another is the sampling search space: instead of the mixed operation as Equation 9, GDAS [4] uses a differentiable sampler (Gumbel-Softmax) to choose an operation between two nodes in a cell:”
- AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “The efficiency of NAS algorithms has been improved by weight sharing [38], differentiable optimization [32], or stochastic sampling [11, 52].”
- A Comprehensive Survey of Neural Architecture Search (2020, methodology): “To this end, GDAS [82] proposes to use a differentiable architecture sampler in a training iteration to sample only one subgraph in the DAG, so only one part of the DAG needs to be optimized in one iteration.”
- DADA: Differentiable Automatic Data Augmentation (2020, positive): “improved if we can relax the optimization to a dierentiable one and jointly optimize network weights and DA parameters in a single-pass way. Motivated by the dierentiable neural architecture search [5,19,26], we propose a Dierentiable Automatic Data Augmentation (DADA) to relax the optimization prob”
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search (2020, methodology): “For each architecture candidate, we can calculate its probability of being a good architecture from the architecture encoding following SETN (Dong & Yang, 2019a).”
- BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “2020) that BANANAS was competitive with nine other popular NAS algorithms, including one-shot methods (Liu, Simonyan, and Yang 2018; Xu et al. 2019; Dong and Yang 2019).”
- Network Pruning via Transformable Architecture Search (2019, methodology): “Differentiable methods [6, 28, 3] dramatically decrease the computation costs but they usually assume that the number of channels in different searching candidates is the same.”
- EENA: Efficient Evolution of Neural Architecture (2019, neutral): “33 GDAS(FRC) [9] 2.”
- Single Path One-Shot Neural Architecture Search with Uniform Sampling (2019, methodology): “s slightly better. So we use it by default in this paper. We note that sampling a path according to architecture distribution during optimization is already used in previous weight sharing approaches [22,4,31,28,6,20]. The dierence is that, the distribution (A) is a xed priori during our training (E”
- Contrastive Adaptation Network for Unsupervised Domain Adaptation (2019, background): “y against the state-of-the-art methods and produces more discriminative features. 1. Introduction Recent advancements in deep neural networks have successfully improved a variety of learning problems [40,8, 26,19,20]. For supervised learning, however, massive labeled training data is still the key t”
- BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, methodology): “To avoid repeated training of candidate networks, weight-sharing NAS methods [7, 43, 18, 1, 6, 13, 82] optimize a supernet that encodes the whole search space, then rate each candidate architecture according to its weights inherited from the supernet.”
- Neural Ensemble Search for Performant and Calibrated Predictions (2020, background): “A recent focus in NAS research has been on computational efficiency with algorithms utilizing gradient-based optimization [8, 12, 36, 55, 59, 56], network morphisms [7, 13, 14], or multi-fidelity optimization [3, 17, 60].”
- Improving One-Shot NAS by Suppressing the Posterior Fading (2019, background): “[11] develops a differentiable sampler over the search space to achieve impressive speed.”
- Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search (2019, methodology): “DARTS [18] and many of its variants [29, 30, 8, 5] are also a nested optimization based on weight-sharing but in a differentiable way.”
- Propagation Model Search for Graph Neural Networks (2020, background): “While ω ∗ ( α ) is coupled with α in (6), such an issue can be empirically addressed by approximating ω ∗ ( α ) with the current ω (Cai, Zhu, and Han 2019; Liu, Simonyan, and Yang 2019; Yao et al. 2020b; Dong and Yang 2019).”
- DARTS-: Robustly Stepping out of Performance Collapse Without Indicators (2020, background): “Since then, many subsequent works have been dedicated to accelerating the process (Dong & Yang, 2019b), reducing memory cost (Xu et al., 2020), or fostering its ability such as hardware-awareness (Cai et al., 2019; Wu et al., 2019), finer granularity (Mei et al., 2020) and so on.”