Tunability: Importance of Hyperparameters of Machine Learning Algorithms

By Philipp Probst, Anne-Laure Boulesteix, Bernd Bischl. 2018

816 citations recorded by the source corpus.

Modern supervised machine learning algorithms involve hyperparameters that have to be set before running them. Options for setting hyperparameters are default values from the software package, manual configuration by the user or configuring them for optimal predictive performance by a tuning procedure. The goal of this paper is two-fold. Firstly, we formalize the problem of tuning from a statistical point of view, define data-based defaults and suggest general measures quantifying the tunability of hyperparameters of algorithms. Secondly, we conduct a large-scale benchmarking study based on 38 datasets from the OpenML platform and six common machine learning algorithms. We apply our measures to assess the tunability of their parameters. Our results yield default values for hyperparameters and enable users to decide whether it is worth conducting a possibly time consuming tuning strategy, to focus on the most important hyperparameters and to chose adequate hyperparameter spaces for tuning.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “Meta-learning can be used to assess the importance of single hyperparameters allowing to remove unimportant hyperparameters from the configuration space (Hutter et al., 2014; Wistuba et al., 2015a; van Rijn & Hutter, 2018; Probst et al., 2019) or identify promising regions (Wistuba et al., 2015b).”
Survey on Automated Machine Learning (2019, positive): “Meta-learning can be used to asses the importance of single hyperparameters allowing to remove unimportant hyperparameters from the configuration space (Hutter et al., 2014; van Rijn and Hutter, 2018; Probst et al., 2019).”
Meta-Learning: A Survey (2018, neutral): “In [120], this was done using about 500,000 OpenML experiments on 6 algorithms and 38 datasets.”
Few-shot learning with adaptively initialized task optimizer: a practical meta-learning approach (2019, background): “2017), and hyper-parameter tuning (Franceschi et al. 2017; Probst et al. 2019).”
Random forest versus logistic regression: a large-scale benchmark experiment (2018, background): “The performance of RF is known to be relatively robust against parameter specifications: performance generally depends less on parameter values than for other machine learning algorithms [19].”
Optimizer Benchmarking Needs to Account for Hyperparameter Tuning (2019, background): “Probst et al. (2019) deﬁne tunability of an ML algorithm as the performance difference between a reference conﬁguration (e.g., the default hyperparameters of the algorithm) and the best possible conﬁguration on each dataset.”
Hyperparameter Tuning for Machine Learning Algorithms Used for Arabic Sentiment Analysis (2021, background): “A comprehensive study on tunability is given in [14]; the authors constructed the tuning problem in statistical terms as well as suggesting tunability quantifying measures of algorithms’ hyperparameters.”
Grid search in hyperparameter optimization of machine learning models for prediction of HIV/AIDS test results (2021, background): “[18] presented the tunability importance of hyperparameter of machine learning.”
A random forest based approach for predicting spreads in the primary catastrophe bond market (2020, background): “There are certain default values that have been suggested following empirical experiments on various data sets but one can use an optimising tuning strategy with respect to prediction performance to select the most suitable values specifically for the data set under study (Probst et al. 2018).”
Hyperparameters and tuning strategies for random forest (2018, methodology): ““Tunability” is defined as the amount of performance gain compared with default values that can be achieved by tuning one hyperparameter (“tunability” of the hyperparameter) or all hyperparameters (“tunability” of the algorithm); see Probst et al. (2018) for more details.”
How to Train Your MAML to Excel in Few-Shot Classification (2021, methodology): “Meta-learning has been applied in various fields, including imbalance learning [74, 106], data compression [103], architecture search [26], recommendation systems [96], data augmentation [69], teaching [27], and hyper-parameter tuning [33, 66].”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “However, it is often the case that some HPCs work relatively well on a large variety of data sets (Probst et al., 2019).”