Scalable Bayesian Optimization Using Deep Neural Networks

By Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Rajagopalan Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat, Ryan P. Adams. 2015

1,144 citations recorded by the source corpus.

Bayesian optimization is an effective methodology for the global optimization of functions with expensive evaluations. It relies on querying a distribution over functions defined by a relatively cheap surrogate model. An accurate model for this distribution over functions is critical to the effectiveness of the approach, and is typically fit using Gaussian processes (GPs). However, since GPs scale cubically with the number of observations, it has been challenging to handle objectives whose optimization requires many evaluations, and as such, massively parallelizing the optimization. In this work, we explore the use of neural networks as an alternative to GPs to model distributions over functions. We show that performing adaptive basis function regression with a neural network as the parametric form performs competitively with state-of-the-art GP-based approaches, but scales linearly with the number of data rather than cubically. This allows us to achieve a previously intractable degree of parallelism, which we apply to large scale hyperparameter optimization, rapidly finding competitive models on benchmark object recognition tasks using convolutional networks, and image caption generation using neural language models.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “Finally, previous work has explored building ensembles of models produced by hyperparameter searches (Snoek et al., 2015; Mendoza et al., 2016; Saikia et al., 2020), including greedy selection strategies (Caruana et al.”
Learning Transferable Architectures for Scalable Image Recognition (2017, methodology): “rovide a viable search strategy, RL does signiﬁcantly improve our ability to learn neural architectures. 4. Related Work The proposed method is related to previous work in hyperparameter optimization [43, 4, 5, 53, 54, 6, 39] – especially recent approaches in designing architectures such as Neural F”
MobileNetV2: Inverted Residuals and Linear Bottlenecks (2018, background): “Recently there has been lots of progress in algorithmic architecture exploration included hyperparameter optimization [9, 10, 11] as well as various”
Advances and Open Problems in Federated Learning (2019, background): “Hyperparameter optimization (HPO) has a long history under the framework of AutoML [339, 237, 241], but it mainly concerns how to improve the model accuracy [59, 364, 321, 159] rather than communication and computing efficacy for”
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, methodology): “[70] J.”
BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “Alternative models, such as random forests (Hutter et al., 2011) or Bayesian neural networks (Snoek et al., 2015; Springenberg et al., 2016; Perrone et al., 2017), scale better with the number of dimensions, but with the exception of Klein et al. (2017c) have not yet been adopted for multi-ﬁdelity…”
AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “Bayesian methods can make the sampling procedure in random search more efficient [25, 43, 44].”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “Currently, BO has been applied for hyper-parameter tuning in deep learning networks, which has led to achievements in both image classification (Snoek et al., 2015) and natural language processing (Melis et al.”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “There is also prior work on using neural network models in BO for hyperparameter optimization (Snoek et al. 2015; Springenberg et al. 2016), The explicit goal of these papers is to improve the efficiency of Gaussian process-based BO from cubic to linear time, not to develop a different type of…”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, background): “Bayesian optimization has received huge attention from the machine learning community in tuning deep neural networks for different tasks including classification tasks [113, 112], speech recognition [24] and natural language processing [81].”
NAS-Bench-101: Towards Reproducible Neural Architecture Search (2019, methodology): “While Bayesian optimization methods based on Bayesian neural networks (Snoek et al., 2015; Springenberg et al., 2016) are generally applicable to this benchmark, we found their computational overhead compared to the other methods to be prohibitively expensive for an exhaustive empirical evaluation.”
Efficient Progressive Neural Architecture Search (2018, methodology): “Typical examples are Bayesian optimization [23, 24] and sequential model-based optimization (SMBO) [15].”
MaskConnect: Connectivity Learning by Gradient Descent (2018, neutral): “However, most of this prior work falls within the genre of hyper-parameter optimization [13,14,15] rather than architecture or connectivity learning.”
Efficient Architecture Search by Network Transformation (2017, neutral): “ - 7.72 All-CNN [26] - - 7.25 ResNet [10] 110 1.7M 6.61 Tree+Max-Avg [16] - - 6.05 Wide ResNet [34] 16 11.0M 4.81 DenseNet [11] 190 25.6M 3.46 MetaQNN [4] 8 11.18M 6.92 Scalable Bayesian Optimization [25] - - 6.37 NAS with strides [35] 20 2.5M 6.01 Large-Scale Evolution [20] - 5.4M 5.40 NAS with max”
A genetic programming approach to designing convolutional neural network architectures (2017, methodology): “[29] used a deep neural network instead of the Gaussian process to reduce the computational cost for the surrogate model building and succeeded in improving the scalability.”