Multiple Adaptive Bayesian Linear Regression for Scalable Bayesian Optimization with Warm Start

By Valerio Perrone, Rodolphe Jenatton, Matthias W. Seeger, Cédric Archambeau. 2017

25 citations recorded by the source corpus.

Bayesian optimization (BO) is a model-based approach for gradient-free black-box function optimization. Typically, BO is powered by a Gaussian process (GP), whose algorithmic complexity is cubic in the number of evaluations. Hence, GP-based BO cannot leverage large amounts of past or related function evaluations, for example, to warm start the BO procedure. We develop a multiple adaptive Bayesian linear regression model as a scalable alternative whose complexity is linear in the number of observations. The multiple Bayesian linear regression models are coupled through a shared feedforward neural network, which learns a joint representation and transfers knowledge across machine learning problems.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “Warm-Started Multitask Learning uses the set of prior evaluations P to learn a joint task representation [97] which is used to train surrogate models on prior tasks tj and integrate them to a feed-forward neural network to learn a joint task representation that can predict accurately a set of evalua”
Meta-Learning: A Survey (2018, neutral): “One can then learn how to combine these per-task predictions to warm-start or guide optimization techniques on a new task tnew [45, 114, 161, 187], as discussed in Sect.”
Continual Learning in Practice (2019, background): “The problem can be lifted to warm-starting using Hyper-Parameter Optimization (HPO), e.g. by Bayesian Optimization [8], for which scalable methods exist whose complexity is linear in the number of observations [9].”
Transferable AutoML by Model Sharing Over Grouped Datasets (2019, background): “The work [30] learns model similarity by building a shared (multi-task) representation for the hyper-parameter space.”
Amazon SageMaker Autopilot: a white box AutoML solution at scale (2020, background): “Knowledge of optimal HPs could also be transferred across different datasets in future work (Perrone et al., 2017; 2018; 2019b; Salinas et al., 2020).”
Probabilistic Matrix Factorization for Automated Machine Learning (2017, methodology): “Perrone et al. (2017) also effectively learn a task-speciﬁc embedding, but instead of using Bayesian neural networks end-to-end like in (Springenberg et al., 2016), they use feed-forward neural networks to learn the basis functions of a Bayesian linear regression model.”
Auto-Differentiating Linear Algebra (2017, methodology): “[17] use our work to scale up Bayesian optimization, using a hybrid setup with neural networks and Bayesian inference.”