On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
By Li Yang, Abdallah Shami. Neurocomputing, 2020
2,874 citations recorded by the source corpus.
Abstract Machine learning algorithms have been used widely in various applications and areas. To fit a machine learning model into different problems, its hyper-parameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model’s performance. It often requires deep knowledge of machine learning algorithms and appropriate hyper-parameter optimization techniques. Although several automatic optimization techniques exist, they have different strengths and drawbacks when applied to different types of problems. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization. This survey paper will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively.
Subfields: Artificial Intelligence, Computational Theory and Mathematics
View Paper: https://www.semanticscholar.org/paper/2e5d2f2dc01b150dffc163a9f457848e9b5b5c38
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Anne-Laure Boulesteix
- Bernd Bischl
- Abdallah Moubayed
- Marius Thomas Lindauer
- Michel Lang
- Martin Binder
- Li Yang
- Abdallah Shami
In Computational Theory and Mathematics
- Abdallah Moubayed
- Anne-Laure Boulesteix
- Li Yang
- Abdallah Shami
- Tobias Pielok
- Martin Binder
- Michel Lang
- Jakob Richter
Citations in the corpus
- MTH-IDS: A Multitiered Hybrid Intrusion Detection System for Internet of Vehicles (2021, methodology): “Additionally, the main hyper-parameter of k-means, the number of clusters [45], k , is tuned by Bayesian Optimization (BO) to improve the quality of the subset.”
- Hyperparameter Tuning for Machine Learning Algorithms Used for Arabic Sentiment Analysis (2021, background): “Moreover, PSO is easy to parallelize since particles operate independently and only need to share information with each other after each iteration [36].”
- Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, background): “Yang and Shami (2020) provide a very high-level overview of search spaces, HPO techniques, and tools.”
- Towards Unbounded Machine Unlearning (2023, background): “Furthermore, Goel et al. [2022] argue that it’s not a necessary condition either: retraining from scratch may arrive at different model distributions due to changes in e.g. hyperparameters [Yang and Shami, 2020].”