Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn

By Brent Komer, James Bergstra, Chris Eliasmith. 2014

317 citations recorded by the source corpus.

Hyperopt-sklearn is a new software project that provides automatic algorithm configuration of the Scikit-learn machine learning library. Following Auto-Weka, we take the view that the choice of classifier and even the choice of preprocessing module can be taken together to represent a single large hyper- parameter optimization problem. We use Hyperopt to define a search space that encompasses many standard components (e.g. SVM, RF, KNN, PCA, TFIDF) and common patterns of composing them together. We demonstrate, using search algorithms in Hyperopt and standard benchmarking data sets (MNIST, 20-Newsgroups, Convex Shapes), that searching this space is practical and effective. In particular, we improve on best-known scores for the model space for both MNIST and Convex Shapes.

Subfields: Artificial Intelligence, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In ML Systems & Hardware

Citations in the corpus

Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “HyperOpt (Komer et al., 2014) realizes asynchronous parallelization and uses TPE to facilitate the HPO.”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “hyperopt-Sklearn [64] is another AutoML framework which is based on Scikit-learn machine learning library.”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “Hyperopt-Sklearn hyperopt-sklearn or hpsklearn (Komer et al., 2014) is a framework for fitting classification and regression pipelines based on hyperopt.”
Survey on Automated Machine Learning (2019, methodology): “hyperopt-sklearn (Komer et al., 2014) is a framework for fitting classification and regression pipelines.”
Efficient Benchmarking of Hyperparameter Optimizers via Surrogates (2015, methodology): “…2011), which have been successfully applied to problems ranging from deep neural networks to combined model selection and hyperparameter optimization (Bergstra et al. 2011; Snoek, Larochelle, and Adams 2012; Thornton et al. 2013; Komer, Bergstra, and Eliasmith 2014; Bergstra, Yamins, and Cox 2013).”
Initializing Bayesian Hyperparameter Optimization via Meta-Learning (2015, methodology): “Here, we evaluated MI-SMBO on small and mediumsized hyperparameter optimiziation problems, and an important open research question is to extend it to even larger configuration spaces, such as those of Auto-WEKA (Thornton et al. 2013) and Hyperopt-Sklearn (Komer, Bergstra, and Eliasmith 2014).”
Hopular: Modern Hopfield Networks for Tabular Data (2022, methodology): “For LightGBM we use the default hyperparameter ranges as specified by hyperopt-sklearn (Komer et al., 2014).”
A quantitative discriminant method of elbow point for the optimal number of clusters in clustering algorithm (2020, methodology): “Our newly proposed method is based on Elbow method[15], K-means++[16], MinMaxScaler [17] for normalization, and cosine of interaction angle of elbow as criteria.”
On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice (2020, background): “hyperopt-sklearn [111] and hyperas [112] are the two libraries that can apply Hyperopt to scikit-learn and Keras libraries.”
Surrogate Benchmarks for Hyperparameter Optimization (2014, methodology): “The state of the art in practical optimization of hyperparameters is defined by Bayesian optimization methods [16, 29, 2], which have been successfully applied to problems ranging from deep neural networks to combined model selection and hyperparameter optimization [2, 29, 30, 19, 5].”
A brief Review of the ChaLearn AutoML Challenge: Any-time Any-dataset Learning without Human Intervention (2016, methodology): “…Algorithm Configuration (SMAC) (Hutter et al., 2011) and hyperopt (Bergstra et al., 2013), combined with Weka (the Auto-WEKA software (Thornton et al., 2013)) or scikit-learn (the hyperopt-sklearn software (Komer et al., 2014)) were promising candidate solutions for the AutoML challenge.”
Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning (2020, methodology): “Later systems started employing model-based global optimization algorithms, such as AutoWEKA (Thornton et al., 2013; Kotthoff et al., 2019), which is built around the WEKA software (Hall et al., 2009) and SMAC (Hutter et al., 2011) and uses cross-validation with racing for model evaluation, and Hype”
ML-Plan for Unlimited-Length Machine Learning Pipelines (2018, background): “Various state-of-the-art AutoML tools (Thornton et al., 2013; Komer et al., 2014; Feurer et al., 2015; Olson and Moore, 2016; de Sá et al., 2017) have shown impressive results in selecting machine learning algorithms and optimizing their hyperparameters to form a machine learning pipeline (ML pipeli”
Hyperopt: a Python library for model selection and hyperparameter optimization (2015, methodology): “[Ber14]), but our development and testing efforts have focused on the optimization of hyperparameters for deep neural networks [hp-dbn], convolutional neural networks for object recognition [hp-convnet], and algorithms within the Scikit-Learn library ([Kom14] and this paper).”
A review of automatic selection methods for machine learning algorithms and hyper-parameter values (2016, methodology): “Automatic selection can save effort for machine learning tool users, reduce the machine learning skill required of users, and improve model accuracy (Komer et al. 2014; Snoek et al. 2012; Bergstra et al. 2011).”
Automated Machine Learning: Methods, Systems, Challenges (2019, unknown): “[30] extended the HPO problem to the Full Model Selection problem, which includes selecting a preprocessing algorithm, a feature selection algorithm, a classifier and all their hyperparameters.”
Automating model search for large scale machine learning (2015, methodology): “Existing techniques to automate the search for a high-quality predictive model focus on the single node setting; extensions for large-scale problems or distributed settings are very basic [28].”
Efficient and Robust Automated Machine Learning (2015, unknown): “[11]) performs best in low-dimensional problems with numerical hyperparameters, tree-based models have been shown to be more successful in highdimensional, structured, and partly discrete problems [12] – such as the CASH problem – and are also used in the AutoML framework hyperopt-sklearn [13].”
Towards Further Automation in AutoML (2018, methodology): “, 2013), hyperopt-sklearn (Komer et al., 2014) and Auto-sklearn (Feurer et al.”