Multi-fidelity Bayesian Optimisation with Continuous Approximations

By Kirthevasan Kandasamy, Gautam Dasarathy, Jeff Schneider, Barnab'as P'oczos. 2017

268 citations recorded by the source corpus.

Bandit methods for black-box optimisation, such as Bayesian optimisation, are used in a variety of applications including hyper-parameter tuning and experiment design. Recently, multi-fidelity methods have garnered considerable attention since function evaluations have become increasingly expensive in such applications. Multi-fidelity methods use cheap approximations to the function of interest to speed up the overall optimisation process. However, most multi-fidelity methods assume only a finite number of approximations. On the other hand, in many practical applications, a continuous spectrum of approximations might be available. For instance, when tuning an expensive neural network, one might choose to approximate the cross validation performance using less data N and/or few training iterations T. Here, the approximations are best viewed as arising out of a continuous two dimensional space (N, T). In this work, we develop a Bayesian optimisation method, BOCA, for this setting. We characterise its theoretical properties and show that it achieves better regret than than strategies which ignore the approximations. BOCA outperforms several other baselines in synthetic and real experiments.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Probabilistic & Bayesian Methods, Mathematical Optimization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Probabilistic & Bayesian Methods

In Mathematical Optimization

Citations in the corpus

BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “, 2016), and using multiple fidelities (Swersky et al., 2013; Kandasamy et al., 2017).”
Multi-Fidelity Automatic Hyper-Parameter Tuning via Transfer Series Expansion (2019, methodology): “Multi-ﬁdelity optimization has been deeply studied on surrogate model optimization such as Bayesian optimization (Huang et al. 2006; Kandasamy et al. 2016; 2017).”
Random Search and Reproducibility for Neural Architecture Search (2019, background): “In traditional hyperparameter optimization, it is standard for empirical results to be reported over 10 independent experimental runs [14, 25, 23, 27].”
Neural Architecture Search with Bayesian Optimisation and Optimal Transport (2018, methodology): “BO methods for early stopping in iterative training procedures [17, 19] can be easily incorporated by defining a fidelity space.”
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (2016, methodology): “We note that the algorithm of Figure 9 was originally proposed by Karnin et al. (2013) for the stochastic setting.”
Bayesian Optimization for Adaptive Experimental Design: A Review (2020, methodology): “Recently, [115] proposed BO for an optimisation problem with multi-fidelity data.”
Practical Hyperparameter Optimization for Deep Learning (2018, methodology): “To obtain good anytime performance, a recent trend in BO is to extend the traditional blackbox setting by exploiting cheaper fidelities of the objective function (Swersky et al., 2014; Klein et al., 2017a; Swersky et al., 2013; Kandasamy et al., 2017).”
Supplementary material for DEHB: Evolutionary Hyperband for Scalable, Robust and Efﬁcient Hyperparameter Optimization (2021, methodology): “Dragonﬂy implements BOCA (Kandasamy et al. [2017]) which performs BO with low-cost approximations of function evaluations on ﬁdelities treated as a continuous domain.”
Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization (2019, methodology): “Indeed, we can observe that the main assumption that “low-cost samples give a representative view of the space” [56] does not fully hold.”
Multi-fidelity Gaussian Process Bandit Optimisation (2016, methodology): “In follow up work (Kandasamy et al., 2017), we extend multi-fidelity optimisation to settings with continuous approximations.”
Multi-Fidelity Black-Box Optimization with Hierarchical Partitions (2018, methodology): “Also, we perform our experiments in a noiseless setting and therefore no Gaussian noise is added to the function evaluations, unlike in (Kandasamy et al., 2017).”
Recent Advances in Bayesian Optimization (2022, background): “In a follow-up work [91], MF-GP-UCB is extended to the continuous idelity space.”