Hyper-Parameter Optimization: A Review of Algorithms and Applications

By Tong Yu, Hong Zhu. 2020

690 citations recorded by the source corpus.

Since deep neural networks were developed, they have made huge contributions to everyday lives. Machine learning provides more rational advice than humans are capable of in almost every aspect of daily life. However, despite this achievement, the design and training of neural networks are still challenging and unpredictable procedures. To lower the technical thresholds for common users, automated hyper-parameter optimization (HPO) has become a popular topic in both academic and industrial areas. This paper provides a review of the most essential topics on HPO. The first section introduces the key hyper-parameters related to model training and structure, and discusses their importance and methods to define the value range. Then, the research focuses on major optimization algorithms and their applicability, covering their efficiency and accuracy especially for deep learning networks. This study next reviews major services and toolkits for HPO, comparing their support for state-of-the-art searching algorithms, feasibility with major deep learning frameworks, and extensibility for new modules designed by users. The paper concludes with problems that exist when HPO is applied to deep learning, a comparison between optimization algorithms, and prominent approaches for model evaluation with limited computational resources.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Industrial and Manufacturing Engineering, Optimization & Training, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Junjie Yang
  2. Kaiyi Ji
  3. Yingbin Liang
  4. Wei Ye
  5. Jindong Wang
  6. Zhengran Zeng
  7. Xingxu Xie
  8. Shikun Zhang

In Computational Theory and Mathematics

  1. Jindong Wang
  2. Junjie Yang
  3. Gaurav Menghani
  4. Kaiyi Ji
  5. Yingbin Liang

In Industrial and Manufacturing Engineering

  1. Junjie Yang
  2. Gaurav Menghani
  3. Kaiyi Ji
  4. Yingbin Liang

In Optimization & Training

  1. Jindong Wang
  2. Hao Chen
  3. Yidong Wang
  4. Junjie Yang
  5. Gaurav Menghani
  6. Kaiyi Ji
  7. Yingbin Liang

In Neural Architecture Search & AutoML

  1. Junjie Yang
  2. Gaurav Menghani
  3. Kaiyi Ji
  4. Yingbin Liang

Citations in the corpus