Freeze-Thaw Bayesian Optimization

By Kevin Swersky, Jasper Snoek, Ryan P. Adams. 2014

293 citations recorded by the source corpus.

In this paper we develop a dynamic form of Bayesian optimization for machine learning models with the goal of rapidly finding good hyperparameter settings. Our method uses the partial information gained during the training of a machine learning model in order to decide whether to pause training and start a new model, or resume the training of a previously-considered model. We specifically tailor our method to machine learning problems by developing a novel positive-definite covariance kernel to capture a variety of training curves. Furthermore, we develop a Gaussian process prior that scales gracefully with additional temporal observations. Finally, we provide an information-theoretic framework to automate the decision process. Experiments on several common machine learning models show that our approach is extremely effective in practice.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “…algorithms, recent methods in Bayesian optimization try to extend the traditional blackbox setting by exploiting cheaper ﬁdelities of the objective function (Swersky et al., 2014; Klein et al., 2017a; Swersky et al., 2013; Kan-dasamy et al., 2017; Klein et al., 2017c; Poloczek et al., 2017).”
Hyperparameter optimization with approximate gradient (2016, background): “Swersky et al. (2014) proposes an approach in which the inner optimization is “freezed” whenever the method decides that the current hyperparameter values are not promising.”
Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor (2020, methodology): “The representatives of these two types are the FBO algorithm [29] and”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “Freeze-thaw Bayesian optimization (Swersky et al., 2014) utilizes a forecast curve to determine whether to freeze a current trial or thaw a previous one.”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “By considering multiple configurations in an iterative procedure simultaneously, the most promising configuration can be optimized in each step (Swersky et al., 2014).”
Survey on Automated Machine Learning (2019, methodology): “A more elaborate approach is Freeze-Thaw Bayesian optimization (Swersky et al., 2014).”
TAPAS: Train-less Accuracy Predictor for Architecture Search (2018, background): “Prediction is obtained either from partial learning curves [7, 11, 2, 18], or from a database of trained experiments [4].”
Peephole: Predicting Network Performance Before Training (2017, methodology): “Some previous methods on performance prediction were developed in the context of hyperparameter optimization, using techniques like Gaussian Process [22] or Last-SeenValue heuristics [16].”
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, methodology): “The poor scaling of naïve methods like grid search with dimensionality has driven interest in more sophisticated hyperparameter optimization methods over the past years [2, 3, 4, 5, 6, 7, 8, 9, 10, 11].”
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization (2016, methodology): “Hyperband is a general-purpose technique that makes minimal assumptions unlike prior conﬁguration evaluation approaches (Domhan et al., 2015; Swersky et al., 2014; Gy¨orgy and Kocsis, 2011; Agarwal et al., 2011; Sparks et al., 2015; Jamieson and Talwalkar, 2015).”
Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, methodology): “An attempt to incorporate this into the Bayesian optimization framework is given in [149].”
Population Based Training of Neural Networks (2017, background): “…process and to better explore the space of promising hyperparameters (Gy¨orgy & Kocsis, 2011; Agarwal et al., 2011; Sabharwal et al., 2016; Swersky et al., 2013, 2014; Domhan et al., 2015; Klein et al., 2016; Snoek et al., 2015; Springenberg et al., 2016). runs to be completed…”
Google Vizier: A Service for Black-Box Optimization (2017, methodology): “While prior work on automated early stopping used Bayesian parametric regression [9, 30], we opted for a Bayesian non-parametric regression, specifically a Gaussian process model with a carefully designed kernel that measures similarity between performance curves.”
Non-stochastic Best Arm Identification and Hyperparameter Optimization (2015, background): “In contrast, Bayesian optimization advocates choosing hyperparameter settings adaptively, but with the exception of Swersky et al. (2014), allocates a fixed budget to each selected hyperparameter setting (Snoek et al., 2012; 2014; Hutter et al., 2011; Bergstra et al., 2011; Bergstra & Bengio, 2012).”
Meta-Learning: A Survey (2018, neutral): “Learning curves are used extensively to speed up hyperparameter optimization on a given task (Kohavi and John, 1995; Provost et al., 1999; Swersky et al., 2014; Chandrashekaran and Lane, 2017).”
Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn (2014, background): “Techniques for recognizing bad performers early could speed up search enormously [Swe14], [Dom14].”
A System for Massively Parallel Hyperparameter Tuning (2018, methodology): “However, freeze-thaw Bayesian optimization (Swersky et al., 2014), which was speciﬁcally designed for models that use iterations as the resource, is known to perform poorly on deep learning tasks (Domhan et al., 2015).”
Bayesian Optimization with Robust Bayesian Neural Networks (2016, methodology): “In future work, we plan to implement Freeze-Thaw Bayesian optimization [31] and Bayesian optimization across dataset sizes [32] in our framework, since both of these generate many cheap function evaluations and thus reach the scalability limit of GPs.”
Analysis of the AutoML Challenge Series 2015-2018 (2019, methodology): “The “freeze-thaw” strategy (Swersky et al., 2014) halts training of models that do not look so promising on the basis of the learning curve, but may restart training them at a later point.”