Hyper-Parameter Optimization: A Review of Algorithms and Applications
690 citations recorded by the source corpus.
Since deep neural networks were developed, they have made huge contributions to everyday lives. Machine learning provides more rational advice than humans are capable of in almost every aspect of daily life. However, despite this achievement, the design and training of neural networks are still challenging and unpredictable procedures. To lower the technical thresholds for common users, automated hyper-parameter optimization (HPO) has become a popular topic in both academic and industrial areas. This paper provides a review of the most essential topics on HPO. The first section introduces the key hyper-parameters related to model training and structure, and discusses their importance and methods to define the value range. Then, the research focuses on major optimization algorithms and their applicability, covering their efficiency and accuracy especially for deep learning networks. This study next reviews major services and toolkits for HPO, comparing their support for state-of-the-art searching algorithms, feasibility with major deep learning frameworks, and extensibility for new modules designed by users. The paper concludes with problems that exist when HPO is applied to deep learning, a comparison between optimization algorithms, and prominent approaches for model evaluation with limited computational resources.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Industrial and Manufacturing Engineering, Optimization & Training, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computational Theory and Mathematics
In Industrial and Manufacturing Engineering
In Optimization & Training
In Neural Architecture Search & AutoML
Citations in the corpus
- PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (2023, background): “Hyperparameter Optimization The importance of hyperparameter optimization in machine learning [30, 31, 32, 33, 34, 35], particularly in the context of fine-tuning deep learning language models such as BERT [36] and GPT [37], cannot be ignored.”
- Bilevel Optimization: Convergence Analysis and Enhanced Design (2020, methodology): “Hyperparameter optimization has become increasingly important as a powerful tool in the automatic machine learning (autoML) (Okuno et al., 2018; Yu & Zhu, 2020).”
- Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better (2021, methodology): “An example is hyper-parameter optimization (HPO) [61] where optimizing the hyper-parameters helps increase the accuracy, which could then be then exchanged for a model with lesser parameters.”