Hyperparameter Optimization

By Aaron Klein. 2017

1,336 citations recorded by the source corpus.

Subfields: Optimization & Training, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Neural Architecture Search & AutoML

Citations in the corpus

Quantifying the Carbon Emissions of Machine Learning (2019, background): “…search for hyperparameter optimization [10], and there is much research being done on ways to improve the efﬁciency of hyperparameter optimization [11, 12], which makes it possible to continue choosing the right hyperparameters for new models without incurring superﬂuous computing and energy costs.”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “A large body of prior work has explored learning hyperparameters of standard optimizers (Bergstra et al., 2011; Feurer & Hutter, 2019).”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “Conceptually, HPOs purposes are threefold (Feurer and Hutter, 2019): to reduce the costly menial work of artificial intelligence (AI) experts and lower the threshold of research and development; to improve the accuracy and efficiency of neural network training (Melis et al., 2017); and to make the…”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “Quanming et al. (2018) and Hutter et al. (2018a) cover less steps of the pipeline creation process and do not provide an empirical evaluation of the presented methods.”
Survey on Automated Machine Learning (2019, methodology): “Recent methods are able to reduce the runtime of AutoML procedures from several hours to mere minutes (Hutter et al., 2018).”
A systematic review on overfitting control in shallow and deep neural networks (2021, methodology): “For more different applications of Bayesian techniques for hyper-parameter optimization, including evidence maximization, Bayesian model averaging, slice sampling, and empirical Bayes, see (Feurer and Hutter 2019).”
AutoDispNet: Improving Disparity Estimation With AutoML (2019, background): “Optimizing hyperparameters for dense prediction tasks with vanilla hyperparameter optimization (HPO) [8, 9, 33, 63, 64, 27] is computationally expensive.”
Techniques for Automated Machine Learning (2019, background): “HPO [26]) or an entire machine learning pipeline automatically (AutoPipeline).”
SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization (2021, background): “A more general view on the problem of HPO is called algorithm conﬁguration (AC) (Hutter et al., 2009; Ans´otegui et al., 2009; L´opez-Ib´a˜nez et al., 2016), where the goal is to determine a well-performing robust conﬁguration across a set of problem instances i from a ﬁnite set I : The origin of SM”
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second (2022, background): “Commonly, resource-intensive searches need to be employed to find suitable hyperparameters (Zoph and Le, 2017; Feurer and Hutter, 2019).”
Hyperparameter Ensembles for Robustness and Uncertainty Quantification (2020, methodology): “Such a “row” typically stems from the application of some hyperparameter-tuning techniques [20].”
Learning Discrete Structures for Graph Neural Networks (2019, methodology): “This technique allows to optimize a number of hyperparameters several orders of magnitude greater than classic methods for hyperparameter optimization (Feurer & Hutter, 2018).”
Tree-structured Parzen estimator: Understanding its algorithm components and their roles for better empirical performance (2023, background): “…(Schneider et al., 2020), material discovery (Xue et al., 2016; Li et al., 2017a; Vahid et al., 2018), financial applications (Gonzalvez et al., 2019), and hyperparameter optimization (HPO) of machine learning algorithms (Loshchilov & Hutter, 2016; Chen et al., 2018; Feurer & Hutter, 2019).”
Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine (2023, background): “Concerns about overfitting with studies of foundation model performance are similar to the valid concerns in traditional machine learning with overfitting during the hyperparameter optimization process [8].”
An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization (2020, neutral): “Details refer to Feurer and Hutter (2018).”
Cache Telepathy: Leveraging Shared Resource Attacks to Learn DNN Architectures (2018, background): “Hyper-parameter tuning [14],[15] is the process of searching for the ideal DNN architecture.”