Bayesian Optimization with Robust Bayesian Neural Networks

By Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, Frank Hutter. 2016

510 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

In Neural Architecture Search & AutoML

Citations in the corpus

BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “In this paper, we focus only on combining Hyperband and Bayesian optimization, but we would like to mention that methods improving BO are potentially applicable to BOHB as well, such as meta learning (Swersky et al., 2013; Feurer et al., 2015; Poloczek et al., 2016; Springenberg et al., 2016), activ”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “2011), BOHAMIANN (Springenberg et al. 2016), BONAS (Shi et al.”
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules (2019, methodology): “Other methods incorporate parallelization or use non-bayesian techniques (Li et al., 2016; Golovin et al., 2017; Shah & Ghahramani, 2015; Springenberg et al., 2016; Gonz´alez et al., 2016) but still either require multiple rounds of optimization or large amounts of compute.”
Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization (2019, methodology): “…Figure 5): random search (Bergstra and Bengio, 2012), SMAC (Hutter et al., 2011) 2 , Tree Parzen Estimator (TPE) (Bergstra et al., 2011) 3 , Bohamiann (Springenberg et al., 2016) 4 , Regularized Evolution (Real et al., 2019), Hy-perband (HB) (Li et al., 2017) and BOHB (Falkner et al., 2018) 5 .”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “It supports many standard surrogate models like Gaussian processes or random forests; yet, also uncommon models like Bayesian neural networks (Springenberg et al., 2016).”
Survey on Automated Machine Learning (2019, background): “It supports many standard surrogate models like Gaussian processes or random forests; yet, also uncommon models like Bayesian neural networks (Springenberg et al., 2016).”
NAS-Bench-101: Towards Reproducible Neural Architecture Search (2019, methodology): “While Bayesian optimization methods based on Bayesian neural networks (Snoek et al., 2015; Springenberg et al., 2016) are generally applicable to this benchmark, we found their computational overhead compared to the other methods to be prohibitively expensive for an exhaustive empirical evaluation.”
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, methodology): “We therefore plan to extend this work to other model classes, such as Bayesian neural networks (Neal, 1996; Hernández-Lobato and Adams, 2015; Blundell et al., 2015; Springenberg et al., 2016; Klein et al., 2017), which may lower the computational overhead while having similar predictive quality.”
Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, methodology): “Specifically, we make use of scale adapted SG-HMC [9], an approach to stochastic gradient Hamiltonian Monte Carlo with automatic hyperparameter discovery.”
Population Based Training of Neural Networks (2017, methodology): “For this reason, a number of methods look to speed up the updating of the posterior with information from each optimisation process by performing early stopping, using intermediate losses to predict final performance, and even modelling the entire time and data dependent optimisation process in orde”
Google Vizier: A Service for Black-Box Optimization (2017, background): “Other researchers have recognized this problem as well, and are working to address it [28].”
Meta-Learning: A Survey (2018, neutral): “One can then learn how to combine these per-task predictions to warm-start or guide optimization techniques on a new task t new [45, 114, 160, 186], as discussed in Section 2.2.3.”
Deep Ensembles: A Loss Landscape Perspective (2019, methodology): “A variety of approximations have been developed for Bayesian neural networks, including Laplace approximation (MacKay, 1992), Markov chain Monte Carlo methods (Neal, 1996; Welling & Teh, 2011; Springenberg et al., 2016), variational Bayesian methods (Graves, 2011; Blundell et al.”
Decoupled Certainty-Driven Consistency Loss for Semi-supervised Learning (2019, methodology): “A few uncertainty modelling methods have been proposed based on Bayesian neural network [19, 16, 10, 14, 26].”
Fast Bayesian hyperparameter optimization on large datasets (2017, methodology): “Popular choices for the objective model are Gaussian processes (Snoek et al., 2012) (see Section 2.1), random forests (Hutter et al., 2011) or Bayesian neural networks (Snoek et al., 2015; Springenberg et al., 2016).”
RECIPE: A Grammar-Based Framework for Automatically Evolving Classification Pipelines (2017, methodology): “We identified six different classification models that had their components optimized using this approach: (i) artificial neural networks [8– 10], (ii) rule induction algorithms [11], (iii) support vector machines [12, 13], (iv) decision trees [14], (v) Bayesian network classifiers [15, 16] and (vi)”
On-Demand Deep Model Compression for Mobile Devices: A Usage-Driven Model Selection Framework (2018, methodology): “Various DNN compression techniques have been proposed, including weight compression [6] [16] [25], convolution decomposition [7] [18] [32], and compact architectures [20] [31].”