Google Vizier: A Service for Black-Box Optimization

By D. Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John E. Karro, David Sculley. 2017

846 citations recorded by the source corpus.

Any sufficiently complex system acts as a black box when it becomes easier to experiment with than to understand. Hence, black-box optimization has become increasingly important as systems have become more complex. In this paper we describe Google Vizier, a Google-internal service for performing black-box optimization that has become the de facto parameter tuning engine at Google. Google Vizier is used to optimize many of our machine learning models and other systems, and also provides core capabilities to Google's Cloud Machine Learning HyperTune subsystem. We discuss our requirements, infrastructure design, underlying algorithms, and advanced features such as transfer learning and automated early stopping that the service provides.

Subfields: Artificial Intelligence, Management Science and Operations Research, Optimization & Training, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Optimization & Training

In Evaluation & Benchmarks

Citations in the corpus

Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “re2). Both parameters are human-interpretable such that larger values of N and M increase regularization strength. Standard methods may be employed to efﬁ- ciently perform hyperparameter optimization [45,14], however given the extremely small search space we ﬁnd that naive grid search is quite effec”
Universal Sentence Encoder (2018, methodology): “Transfer task model hyperparamaters are tuned using a combination of Vizier (Golovin et al., 2017) and light manual tuning.”
BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, neutral): “The data gathered by BOHB on different budgets could also be used to quantify the importance of hyperparameters (Hutter et al., 2014; Biedenkapp et al., 2017; Golovin et al., 2017; van Rijn & Hutter, 2018).”
Neural Optimizer Search with Reinforcement Learning (2017, methodology): “Weight decay, embedding, state and output dropout, state reset probability and embedding ratio are tuned with a black-box hyperparam-eter tuner (Golovin et al., 2017), similarly to Melis et al. (2017).”
AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, methodology): “For Random Search and Vizier, we report their results when searching for both learning rate and the weight of L2 penalty.”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “Figure 17: Architecture of Google Vizier (Golovin et al., 2017)”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “BO with Gaussian processes (GPs) has had success in deep learning hyperparameter optimization (Golovin et al. 2017; Falkner, Klein, and Hutter 2018), and is a leading method for efficient zeroth order optimization of expensive-to-evaluate functions in Euclidean spaces.”
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules (2019, methodology): “Other methods incorporate parallelization or use non-bayesian techniques (Li et al., 2016; Golovin et al., 2017; Shah & Ghahramani, 2015; Springenberg et al., 2016; González et al., 2016) but still either require multiple rounds of optimization or large amounts of compute.”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “Various open-source designs for AutoML services have been proposed (e.g., Sparks et al., 2015; Chan, 2017; Swearingen et al., 2017; Koch et al., 2018) but also several commercial solutions exist (e.g., Golovin et al., 2017; Clouder, 2018; H2O.ai, 2018).”
AlphaClean: Automatic Generation of Data Cleaning Pipelines (2019, methodology): “Since Bayesian optimization is inherently sequential, for parallel and distributed settings, the community is increasingly studying randomized and grid search schemes [20, 32, 33].”
Survey on Automated Machine Learning (2019, methodology): “Various open-source designs for AutoML services have been proposed, e.g., (Sparks et al., 2015; Chan, 2017; Swearingen et al., 2017; Koch et al., 2018), but also several commercial solutions exist, e.g., (Golovin et al., 2017; Clouder, 2018; H2O.ai, 2018).”
Random Search and Reproducibility for Neural Architecture Search (2019, methodology): “These methods use Bayesian optimization [26; 24; 13], performance prediction [15; 10], or multi-armed bandits [22; 28; 29] to adaptively allocate resources to different configurations.”
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (2016, methodology): “” Several hybrid methods combining adaptive configuration selection and evaluation have also been introduced (Swersky et al., 2013, 2014; Domhan et al., 2015; Kandasamy et al., 2016; Klein et al., 2017a; Golovin et al., 2017).”
Population Based Training of Neural Networks (2017, methodology): “Many of these Bayesian optimisation approaches can be further sped up by attempting to parallelise them – training independent models in parallel to quicker update the Bayesian model (though potentially introducing bias) (Shah & Ghahramani, 2015; González et al., 2016; Wu & Frazier, 2016; Rasley et ”
Searching for Efficient Multi-Scale Architectures for Dense Image Prediction (2018, methodology): “We ﬁnd that an effective random search policy provides a strong baseline [5, 30] and identify several candidate network architectures for scene labeling.”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, methodology): “We detail the models and implementations used in Appendix A. Hyperparameters were tuned for all methods using Bayesian optimization (Golovin et al., 2017) (except on ImageNet) as detailed in Appendix A.8.”
Meta-Learning: A Survey (2018, neutral): “Golovin et al. [58] assume a sequence order (e.g., time) across tasks.”