Neural Architecture Search with Bayesian Optimisation and Optimal Transport

By Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnab'as P'oczos, Eric P. Xing. 2018

666 citations recorded by the source corpus.

Bayesian Optimisation (BO) refers to a class of methods for global optimisation of a function $f$ which is only accessible via point evaluations. It is typically used in settings where $f$ is expensive to evaluate. A common use case for BO in machine learning is model selection, where it is not possible to analytically model the generalisation performance of a statistical model, and we resort to noisy and expensive training and validation procedures to choose the best model. Conventional BO methods have focused on Euclidean and categorical domains, which, in the context of model selection, only permits tuning scalar hyper-parameters of machine learning algorithms. However, with the surge of interest in deep learning, there is an increasing demand to tune neural network \emph{architectures}. In this work, we develop NASBOT, a Gaussian process based BO framework for neural architecture search. To accomplish this, we develop a distance metric in the space of neural network architectures which can be computed efficiently via an optimal transport program. This distance might be of independent interest to the deep learning community as it may find applications outside of BO. We demonstrate that NASBOT outperforms other alternatives for architecture search in several cross validation based model selection tasks on multi-layer perceptrons and convolutional neural networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Quoc V. Le
  2. Yoshua Bengio
  3. Mohit Bansal
  4. Dacheng Tao
  5. Sergey Levine
  6. Jianfeng Gao
  7. Ruslan Salakhutdinov
  8. Percy Liang

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Yi Yang
  4. Ming-Hsuan Yang
  5. Chunhua Shen
  6. Trevor Darrell
  7. Xiaogang Wang
  8. Alan L. Yuille

In Probabilistic & Bayesian Methods

  1. Frank Hutter
  2. Yash Savani
  3. Colin White
  4. Xia Hu
  5. Qingquan Song
  6. Willie Neiswanger
  7. Ameet Talwalkar
  8. Eric P. Xing

In Neural Architecture Search & AutoML

  1. Frank Hutter
  2. Colin White
  3. Yash Savani
  4. Sam Nolen
  5. Xuanyi Dong
  6. Arber Zela
  7. Xia Hu
  8. Quoc V. Le

Citations in the corpus