Neural Architecture Search with Bayesian Optimisation and Optimal Transport
By Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnab'as P'oczos, Eric P. Xing. 2018
666 citations recorded by the source corpus.
Bayesian Optimisation (BO) refers to a class of methods for global optimisation of a function $f$ which is only accessible via point evaluations. It is typically used in settings where $f$ is expensive to evaluate. A common use case for BO in machine learning is model selection, where it is not possible to analytically model the generalisation performance of a statistical model, and we resort to noisy and expensive training and validation procedures to choose the best model. Conventional BO methods have focused on Euclidean and categorical domains, which, in the context of model selection, only permits tuning scalar hyper-parameters of machine learning algorithms. However, with the surge of interest in deep learning, there is an increasing demand to tune neural network \emph{architectures}. In this work, we develop NASBOT, a Gaussian process based BO framework for neural architecture search. To accomplish this, we develop a distance metric in the space of neural network architectures which can be computed efficiently via an optimal transport program. This distance might be of independent interest to the deep learning community as it may find applications outside of BO. We demonstrate that NASBOT outperforms other alternatives for architecture search in several cross validation based model selection tasks on multi-layer perceptrons and convolutional neural networks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Yoshua Bengio
- Mohit Bansal
- Dacheng Tao
- Sergey Levine
- Jianfeng Gao
- Ruslan Salakhutdinov
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Ming-Hsuan Yang
- Chunhua Shen
- Trevor Darrell
- Xiaogang Wang
- Alan L. Yuille
In Probabilistic & Bayesian Methods
- Frank Hutter
- Yash Savani
- Colin White
- Xia Hu
- Qingquan Song
- Willie Neiswanger
- Ameet Talwalkar
- Eric P. Xing
In Neural Architecture Search & AutoML
Citations in the corpus
- Neural Architecture Optimization (2018, methodology): “One need to pay more efforts in setting good K ( x, x (cid:48) ) in the context of architecture design, bringing additional manual efforts whereas the performance might still be unsatisfactory [21].”
- A Comprehensive Survey of Neural Architecture Search (2020, methodology): “SMBO NASBOT [61] NeurIPS18 8.”
- Real-Time Federated Evolutionary Neural Architecture Search (2020, methodology): “Other techniques such as Bayesian optimization [41] is also a good approach reducing the computational complexity of NAS [42].”
- DSNAS: Direct Neural Architecture Search Without Parameter Retraining (2020, background): “But if the searched architecture is the answer, what is the problem? Most NAS methods claim they are discovering best-performing architecture in the designated space efficiently [3, 7, 9], but what specifically does best-performing mean? Given that retraining is conducted in evaluation stage, ones m”
- On Robustness of Neural Architecture Search Under Label Noise (2020, background): “Currently, the mainstream approaches to achieve NAS include Bayesian optimization (Kandasamy et al., 2018; Jin et al., 2019), reinforcement learning (Zoph and Le, 2017; Cai et al., 2018; Pham et al., 2018; Zoph et al., 2018), evolutionary algorithms (Real et al., 2017, 2019) and gradient-based…”
- BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “Note also that both TS and ITS have advantages when running parallel experiments, since they are stochastic acquisition functions that can be directly applied in the batch BO setting (Kandasamy et al. 2018a).”
- Towards modular and programmable architecture search (2019, background): “Researchers have considered a variety of search algorithms, including reinforcement learning [13], evolutionary algorithms [14, 15], MCTS [16], SMBO [16, 17], and Bayesian optimization [18].”
- Auto-GNN: Neural architecture search of graph neural networks (2019, methodology): “The deep architectures discovered by NAS algorithms have outperformed the handcrafted ones at the domains including image classification [12–21], semantic image segmentation [22], and image generation [23].”
- Evolving Robust Neural Architectures to Defend from Adversarial Attacks (2019, background): “in a given space. To find architectures in a defined search space, a search strategy must be employed to explore it. Some widely used search strategies for AAS are: Random Search, Bayesian Optimization [15], Evolutionary Methods [29],[41],[17],[28], Reinforcement Learning [44], [3], [6], [43],[7], [27”
- Searching for a Robust Neural Architecture in Four GPU Hours (2019, background): “Recently, neural architecture search (NAS) has increasingly attracted the interest of researchers [1, 6, 17, 22, 25, 31, 46].”
- EENA: Efficient Evolution of Neural Architecture (2019, methodology): “[19] derive kernel functions for architecture search spaces in order to use classic GP-based BO method, but compared with others, BO method shows no obvious advantages.”
- NAS-Unet: Neural Architecture Search for Medical Image Segmentation (2019, methodology): “The search algorithms mainly includes heuristic algorithm [19], [21], [32]–[34], reinforcement learning [35]–[38], [38], [39], the Bayesian optimization method [40], [41] and the gradient-based method [20], [42], [43].”
- Evaluating the Search Phase of Neural Architecture Search (2019, methodology): “Bayesian optimization was first introduced to the neural architecture search field by Kandasamy et al. (2018) and Jin et al. (2019).”
- Random Search and Reproducibility for Neural Architecture Search (2019, methodology): “In terms of broad reproducibility, with the exception of NASBOT [24] and DARTS [32], the methods in Table 5 only report the performance of the best found architecture, presumably resulting from a single run of the search process.”
- Auto-Keras: An Efficient Neural Architecture Search System (2018, neutral): “• State-of-the-art Methods: SEAS [11], NASBOT [16].”
- NSGA-NET: A Multi-Objective Genetic Algorithm for Neural Architecture Search (2018, methodology): “Finally, Kandasamy et al. Kandasamy et al. (2018) present a Gaussian process based approach to optimize network architectures, viewing the process through a Bayesian optimization lens.”
- Demystifying Parallel and Distributed Deep Learning (2018, methodology): “Training Distribution Model Consolidation Ensemble Learning [118,153,225] Knowledge Distillation [10,97] Model Averaging: Direct [20,33,174,277] , Elastic [126,159,258,268] , Natural Gradient [11,202] Optimization Algorithms First-Order [43,128,132,147,151,168,235] Second-Order [11,28,57,95,139,171,”