Hyperparameter Importance Across Datasets

By Jan N. van Rijn, Frank Hutter. 2017

287 citations recorded by the source corpus.

With the advent of automated machine learning, automated hyperparameter optimization methods are by now routinely used in data mining. However, this progress is not yet matched by equal progress on automatic analyses that yield information beyond performance-optimizing hyperparameter settings. In this work, we aim to answer the following two questions: Given an algorithm, what are generally its most important hyperparameters, and what are typically good values for these? We present methodology and a framework to answer these questions based on meta-learning across many datasets. We apply this methodology using the experimental meta-data available on OpenML to determine the most important hyperparameters of support vector machines, random forests and Adaboost, and to infer priors for all their hyperparameters. The results, obtained fully automatically, provide a quantitative basis to focus efforts in both manual algorithm design and in automated hyperparameter optimization. The conducted experiments confirm that the hyperparameters selected by the proposed method are indeed the most important ones and that the obtained priors also lead to statistically significant improvements in hyperparameter optimization.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “Meta-learning can be used to assess the importance of single hyperparameters allowing to remove unimportant hyperparameters from the configuration space (Hutter et al., 2014; Wistuba et al., 2015a; van Rijn & Hutter, 2018; Probst et al., 2019) or identify promising regions (Wistuba et al., 2015b).”
Survey on Automated Machine Learning (2019, positive): “Meta-learning can be used to asses the importance of single hyperparameters allowing to remove unimportant hyperparameters from the configuration space (Hutter et al., 2014; van Rijn and Hutter, 2018; Probst et al., 2019).”
Meta-Learning: A Survey (2018, neutral): “In [136], this was explored using 250,000 OpenML experiments with 3 algorithms across 100 datasets.”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “…” values that work well under many conditions, by tuning ranges and HPC prior distributions based on extensive meta-data (Feurer, Klein, et al., 2015a; Gijsbers et al., 2021; Perrone et al., 2019; Pfisterer et al., 2021; Probst et al., 2019; Van Rijn & Hutter, 2018; Wistuba et al., 2015a, 2015b).”
Efficient Architecture Search for Continual Learning (2020, methodology): “In detail, we are inspired by [36] and use the grid search method to generate a set of hyperparameters according to Table I.”
Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning (2019, methodology): “A related idea to learn hyperparameter importance has been explored in [45], where a post-hoc functional ANOVA analysis is used to learn priors over the hyperparameter space.”
Tunability: Importance of Hyperparameters of Machine Learning Algorithms (2018, methodology): “We now compare our method with van Rijn and Hutter (2017).”
Taking Human out of Learning Applications: A Survey on Automated Machine Learning (2018, background): “Existing works of this line make effort to evaluate the importance of configurations [120], [121], or identify promising regions in the search space [122].”
OpenML Benchmarking Suites (2017, background): “…that the taxonomies provided by the community can be misleading, and therefore learn taxonomies to cluster classiﬁers based on predictive behavior. van Rijn and Hutter [2018] and Probst et al. [2019a] used it to quantify the hyperparameter importance of machine learning algorithms, while Probst…”