An Efficient Approach for Assessing Hyperparameter Importance

By Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown. 2014

550 citations recorded by the source corpus.

Subfields: Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Neural Architecture Search & AutoML

Citations in the corpus

BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “The data gathered by BOHB on different budgets could also be used to quantify the importance of hyperparameters (Hutter et al., 2014; Biedenkapp et al., 2017; Golovin et al., 2017; van Rijn & Hutter, 2018).”
Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization (2019, methodology): “We now analyze how the different hyperparameters affect the ﬁnal performance, ﬁrst globally with help of the functional ANOVA (Sobol, 1993; Hutter et al., 2014)”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “Meta-learning can be used to assess the importance of single hyperparameters allowing to remove unimportant hyperparameters from the configuration space (Hutter et al., 2014; Wistuba et al., 2015a; van Rijn & Hutter, 2018; Probst et al., 2019) or identify promising regions (Wistuba et al., 2015b).”
Survey on Automated Machine Learning (2019, neutral): “Meta-learning can be used to asses the importance of single hyperparameters allowing to remove unimportant hyperparameters from the configuration space (Hutter et al., 2014; van Rijn and Hutter, 2018; Probst et al., 2019).”
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks (2017, methodology): “Hutter et al. (2014) describe a method to gain insights into the relative importance of hyperparameters by using random forest predictions with functional ANOVA compositions (Gramacy et al., 2013).”
Meta-Learning: A Survey (2018, neutral): “First, in the functional ANOVA [67] approach, hyperparameters are deemed important if they explain most of the variance in algorithm performance on a given task.”
Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging (2017, background): “A priori it is hard to know which hyperparameters will yield the best performance and the selection of the parameters often makes the difference between mediocre and state-of-the-art performance (Hutter et al., 2014).”
Automated Machine Learning on Big Data using Stochastic Algorithm Tuning (2014, background): “The performance of many machine learning algorithms is highly sensitive to the learning of model structure, parameters and hyperparameters [1].”
Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks (2018, methodology): “Recent results indicate that more sophisticated and automated approaches can find better hyperparameters – and thus achieve better results – than humans (Pinto et al., 2009; Coates et al., 2011; Bergstra et al., 2011; Snoek et al., 2012; Bardenet et al., 2013; Hutter et al., 2014).”
An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization (2020, background): “Future work to improve BOSS may further learn the search space (Perrone et al., 2019) or hyperparameter importance (Hoos and Leyton-Brown, 2014).”
LSTM: A Search Space Odyssey (2015, methodology): “We also provide insights gained about hyperparameters and their interaction using functional ANalysis Of VAriance (fANOVA) [19].”
Towards Reproducible Neural Architecture and Hyperparameter Search (2018, methodology): “To analyze the importance of hyperparameters, assessing the change of the final error with respect to changing a single hyperparameter at a time we used the fANOVA tool developed by Hutter et al. (2014).”
Learning Global Additive Explanations for Neural Nets Using Model Distillation (2018, methodology): “1 ADDITIVE F̂ Global additive explanations have been used to analyze inputs to complex, nonlinear mathematical models and computer simulations (Sobol, 2001), analyze how hyperparameters affect the performance of machine learning algorithms (Hutter et al., 2014), and decompose prediction functions in”
Beyond Manual Tuning of Hyperparameters (2015, methodology): “methods can be used to provide a visualization of how hyperparameters affect performance, and to quantify the importance of individual hyperparameters and of their interaction effects [30, 34].”