Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms
By James Bergstra, Daniel L. K. Yamins, David D. Cox. 2013
808 citations recorded by the source corpus.
Sequential model-based optimization (also known as Bayesian optimization) is one of the most efficient methods (per function evaluation) of function minimization. This efficiency makes it appropriate for optimizing the hyperparameters of machine learning algorithms that are slow to train. The Hyperopt library provides algorithms and parallelization infrastructure for performing hyperparameter optimization (model selection) in Python. This paper presents an introductory tutorial on the usage of the Hyperopt library, including the description of search spaces, minimization (in serial and parallel), and the analysis of the results collected in the course of minimization. The paper closes with some discussion of ongoing and future work.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Frank Hutter
- Marco F. Huber
- Marc-André Zöller
- Xia Hu
- Quoc V. Le
- Yoshua Bengio
- Mohit Bansal
- Katharina Eggensperger
In Computational Theory and Mathematics
- Marco F. Huber
- Marc-André Zöller
- Shuhei Watanabe
- Yaochu Jin
- Gang Luo
- Marius Thomas Lindauer
- Frank Hutter
- Markus Olhofer
In Management Science and Operations Research
- Gang Luo
- Marc-André Zöller
- Marco F. Huber
- Shuhei Watanabe
- Katharina Eggensperger
- Frank Hutter
- Matthias Feurer
- Yaochu Jin
In Neural Architecture Search & AutoML
- Frank Hutter
- Marc-André Zöller
- Marco F. Huber
- Marius Thomas Lindauer
- Katharina Eggensperger
- Matthias Feurer
- Shuhei Watanabe
- James Bergstra
Citations in the corpus
- Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “Hyperopt-Sklearn uses Hyperopt [11] to define the search space over the possible Scikit-Learn main components including the learning and preprocessing algorithms.”
- Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “This concept is captured by sequential model- based optimization (SMBO) (Bergstra et al., 2011; Hutter et al., 2011; Bergstra et al., 2013) displayed in Figure 6.”
- Survey on Automated Machine Learning (2019, methodology): “BOHB replaces this random selection by a SMBO procedure.”
- Auto-Keras: An Efficient Neural Architecture Search System (2018, methodology): “For example, it has been used in hyperparameter tuning for machine learning models [3, 15, 21, 24, 40, 44], in which Bayesian optimization searches among different combinations of hyperparameters.”
- Simplifying Graph Convolutional Networks (2019, methodology): “In addition, we use weight decay and tune this hyperparameter on each dataset using hyperopt (Bergstra et al., 2015) for 60 iterations on the public split validation set.”
- Deep learning for side-channel analysis and introduction to ASCAD database (2019, background): “1 Some libraries (such as hyperopt or hyperas, [8]) could have been tested to automatize the search of accurate hyper-parameters in pre-defined sets.”
- Hyperparameter Optimization and Boosting for Classifying Facial Expressions: How good can a "Null" Model be? (2013, methodology): “The details of this meta-model are described in Bergstra et al. (2013a) and implemented in the hyperopt-convnet software available from http: //github.com/jaberg/hyperopt-convnet .”
- Teacher Guided Architecture Search (2018, methodology): “for their helpful comments and discussions. References [1] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in neural information processing systems, pages 2654–2662, 2014. [2] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectur”
- Could graph neural networks learn better molecular representation for drug discovery? A comparison study of descriptor-based and graph-based models (2020, methodology): “5 x64).[61] Then, in the second stage, in order to alleviate the effect of the randomness of data splitting, 50 independent running with different random seeds for data splitting (training/validation/test=8:1:1) were performed to evaluate each ML model in a more reliable way.”
- Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes (2017, methodology): “TPE is used in the HyperOpt tool (Bergstra et al., 2013).”
- Easy Hyperparameter Search Using Optunity (2014, methodology): “HyperOpt offers random search and sequential model-based optimization (Bergstra et al., 2013).”
- RAPID: Rapid and Precise Interpretable Decision Sets (2019, methodology): “We propose using the Tree-structured Parzen Estimator Approach (TPE) [36] as implemented in [37] to find the values of λ 1 , λ 2 , . . . , λ 7 .”