Hyperparameter Search in Machine Learning

By Marc Claesen, B. De Moor. arXiv.org, 2015

484 citations recorded by the source corpus.

We introduce the hyperparameter search problem in the field of machine learning and discuss its main challenges from an optimization perspective. Machine learning methods attempt to build models that capture some element of interest based on given data. Most common learning algorithms feature a set of hyperparameters that must be determined before training commences. The choice of hyperparameters can significantly affect the resulting model's performance, but determining good values can be complex; hence a disciplined, theoretically sound search strategy is essential.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/0173ca962e4ab3d084c89568345e06f67d3d7efc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice (2020, methodology): “Depending on the scale of data, the model complexity, and available computational resources, the evaluation of each hyper-parameter configuration may cost several minutes, hours, days, or even more [93].”
Fluid directed rigid body control using deep reinforcement learning (2018, methodology): “We choose these parameters using grid-based meta-parameter search, which is well known and widely used in machine learning community to avoid training falling into local minima (see e.g., [Claesen and De Moor 2015] for an introduction).”
Deep Learning for the Radiographic Detection of Periodontal Bone Loss (2019, methodology): “Hyperparameters were systematically tuned via grid search 21 .”
A review of the-state-of-the-art in data-driven approaches for building energy prediction (2020, background): “a pre-defined parameter which affects the running time of feature engineering process and prediction accuracy of the developed data-driven model [59]).”
Application of machine learning to construction injury prediction (2016, methodology): “The choiceoftheob-jective function and of the searching scheme is often dictated by the dimensionality of the parameter space, the computational resources available, and the nature of the ML algorithm [19].”
Particle swarm optimization for hyper-parameter selection in deep neural networks (2017, background): “e model may be parametrized by the hyper-parameters λ, and it is given asM = A(T ;λ) [5].”
Supervised Contrastive Learning (2020, background): “Deep network training is well known to be sensitive to hyper-parameters and a large body of literature is devoted to finding efficient ways to perform hyperparameter tuning [2, 8, 3].”