BayesNAS: A Bayesian Approach for Neural Architecture Search

By Hongpeng Zhou, Minghao Yang, Jun Wang, Wei Pan. 2019

221 citations recorded by the source corpus.

One-Shot Neural Architecture Search (NAS) is a promising method to significantly reduce search time without any separate training. It can be treated as a Network Compression problem on the architecture parameters from an over-parameterized network. However, there are two issues associated with most one-shot NAS methods. First, dependencies between a node and its predecessors and successors are often disregarded which result in improper treatment over zero operations. Second, architecture parameters pruning based on their magnitude is questionable. In this paper, we employ the classic Bayesian learning approach to alleviate these two issues by modeling architecture parameters using hierarchical automatic relevance determination (HARD) priors. Unlike other NAS methods, we train the over-parameterized network for only one epoch then update the architecture. Impressively, this enabled us to find the architecture on CIFAR-10 within only 0.2 GPU days using a single GPU. Competitive performance can be also achieved by transferring to ImageNet. As a byproduct, our approach can be applied directly to compress convolutional neural networks by enforcing structural sparsity which achieves extremely sparse networks without accuracy deterioration.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tie-Yan Liu
2. Tao Qin
3. Xu Tan
4. Colin White
5. Yash Savani
6. Martin Jaggi
7. Quoc V. Le
8. Peter Richtárik

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Yi Yang
4. Alan L. Yuille
5. Tao Qin
6. Xiaogang Wang
7. Zhenguo Li
8. Tie-Yan Liu

In Probabilistic & Bayesian Methods

1. Yash Savani
2. Colin White
3. Willie Neiswanger
4. Tao Qin
5. Claudiu Cristian Musat
6. Frank Hutter
7. James T. Kwok
8. Wujie Wen

In Neural Architecture Search & AutoML

1. Colin White
2. Yash Savani
3. Frank Hutter
4. Willie Neiswanger
5. Hanxiao Liu
6. Zhenguo Li
7. Tao Qin
8. Tie-Yan Liu

Citations in the corpus

• Semi-Supervised Neural Architecture Search (2020, background): “, reinforcement learning [41, 42], evolution [20], gradient optimization [15, 17], Bayesian optimization [39]), and plays an important role in NAS [41, 42, 18, 20, 17, 15, 39].”
• BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “[77] Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan.”
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2019, neutral): “3 gradient-based BayesNAS + cutout (Zhou et al., 2019) 2.”
• Evaluating the Search Phase of Neural Architecture Search (2019, methodology): “Please see (Zhou et al., 2019) for more details.”
• Neural Predictor for Neural Architecture Search (2019, background): “Recent advances have reduced this cost significantly [15, 9, 26, 4, 18, 5, 2, 17, 24, 23, 3], but many of them require nontrivial specialized implementations.”
• SGAS: Sequential Greedy Architecture Search (2019, unknown): “rder) [35] 3.000.14 3.3 0.4 gradient DARTS (2nd order) [35] 2.760.09 3.3 1 gradient SNAS (mild) [58] 2.98 2.9 1.5 gradient ProxylessNAS [8] 2.08 - 4 gradient P-DARTS [9] 2.5 3.4 0.3 gradient BayesNAS [63] 2.810.04 3.4 0.2 gradient PC-DARTS [61] 2.570.07 3.6 0.1 gradient SGAS (Cri.1 avg.) 2.660:24 3.”
• NPENAS: Neural Predictor Guided Evolution for Neural Architecture Search (2022, methodology): “The commonly used search strategies include reinforcement learning (RL) [18], [24]–[26], gradientbased methods [13], [16], [27]–[29], Bayesian optimization [3], [4], evolutionary algorithms, [5]–[9] and predictor-based methods [12], [14], [15], [26].”
• Neural Architecture Search with GBDT (2020, methodology): “Several kinds of methods have been adopted in NAS, including reinforcement learning based [38, 16], evolutionary algorithms based [18, 11], Bayesian methods based [36], gradient based [14, 11] and accuracy predictor based methods [14, 27].”
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS (2020, methodology): “…[24], which finds the model by parameter sharing; (ii) DARTS [18], which applies continuous relaxation for super-network training; (iii) BayesNAS [40], which considers the dependency on architectures; and (iv) ASNG-NAS [1], which proposes a stochastic natural gradient method for the NAS problem.”