Sequential Model-Based Optimization for General Algorithm Configuration

By Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown. 2011

2,949 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Networks and Communications, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Computer Networks and Communications

In Optimization & Training

Citations in the corpus

Learning to Optimize Neural Nets (2017, methodology): “Various kinds of methods have been proposed, such those based on Bayesian optimization (Hutter et al., 2011; Bergstra et al., 2011; Snoek et al., 2012; Swersky et al., 2013; Feurer et al., 2015), random search (Bergstra & Bengio, 2012) and gradient-based optimization (Bengio, 2000; Domke, 2012;…”
True Few-Shot Learning with Language Models (2021, background): “Selection criteria can be optimized automatically, e.g. with bayesian optimization [44–46], evolutionary methods [45, 47, 48], reinforcement learning [49], or gradient descent [50–53].”
Practical Bayesian Optimization of Machine Learning Algorithms (2012, methodology): “[7] have developed sequential model-based optimization strategies for the configuration of satisfiability and mixed integer programming solvers using random forests.”
Progressive Neural Architecture Search (2017, methodology): “Sequential Model Based Optimization (SMBO) [17] improves on MCTS by learning a predictive model, which can be used to decide which nodes to expand.”
LSTM: A Search Space Odyssey (2015, methodology): “While there are other methods to efficiently search for good hyperparameters [43], [44], random search has several advantages for our setting: it is easy to implement, trivial to parallelize, and covers the search space more uniformly, thereby improving the follow-up analysis of hyperparameter impor”
AutoML: A Survey of the State-of-the-Art (2019, methodology): “egories, according to the type of probability model they contain: Gaussian Processes (GPs) [120, 121], Tree Parzen Estimators (TPE) [37], and Random Forests [35].”
Scalable Bayesian Optimization Using Deep Neural Networks (2015, methodology): “Among these, the most popular variants in machine learning are the random forest-based SMAC procedure (Hutter et al., 2011) and the tree Parzen estimator (TPE) (Bergstra et al.”
NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, methodology): “4, parameterization of architectures) and a bagged [6] XGBoost regressor to estimate uncertainty of predictions; • Regularized Evolution (RE) [38] using graph2vec features; • Hyperopt with the Tree-structured Parzen Estimator (TPE); [5]; • SMAC [20] using 10-dimensional graph2vec features.”
AutoHAS: Differentiable Hyper-parameter and Architecture Search (2020, background): “Black-box and multi-fidelity HPO methods have a long standing history [4, 20, 21, 22, 27, 22].”
A Comprehensive Survey of Neural Architecture Search (2020, methodology): “For clarity, we classify NAS according to mainstream search methods: reinforcement learning (RL), evolutionary algorithm (EA), gradient optimization (GO), random search (RS) and sequential model-based optimization (SMBO).”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “In addition to GP, random forests (Hutter et al., 2011) are another popular type of surrogate model with a structure similar to a decision tree.”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, methodology): “The length and difficulty of ML research prompted a new field, named AutoML, that aims to automate such progress by spending machine compute time instead of human research time (Fahlman & Lebiere, 1990; Hutter et al., 2011; Finn et al., 2017).”
Bayesian Neural Architecture Search using A Training-Free Performance Metric (2020, methodology): “Efficient Global Optimization) [30, 21] algorithm has been applied extensively for automated algorithm configuration tasks [20, 2, 19].”
Towards modular and programmable architecture search (2019, methodology): “Hyperparameter optimization systems (e.g. Hyperopt [2], Spearmint [3], SMAC [5, 8] and BOHB [4]) are general-purpose and domain-independent.”
Multi-Fidelity Automatic Hyper-Parameter Tuning via Transfer Series Expansion (2019, methodology): “For examples, Bayesian optimization models the search space by Gaussian process (GP) or tree-structure (Hutter, Hoos, and Leyton-Brown 2011).”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “Random forests are used by the Sequential Model-based Algorithm Configuration (SMAC) library [51].”
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules (2019, background): “There has been much previous work to well-tune hyperparameters, especially in Bayesian Optimization (Srinivas et al., 2009; Bergstra et al., 2011; Snoek et al., 2012; Hutter et al., 2011), which are sequential in nature and expensive computationally.”