Autotune: A Derivative-free Optimization Framework for Hyperparameter Tuning

By Patrick N. Koch, Oleg Golovidov, Steven Gardner, Brett Wujek, J. Griffin, Yan Xu. 2018

98 citations recorded by the source corpus.

Machine learning applications often require hyperparameter tuning. The hyperparameters usually drive both the efficiency of the model training process and the resulting model quality. For hyperparameter tuning, machine learning algorithms are complex black-boxes. This creates a class of challenging optimization problems, whose objective functions tend to be nonsmooth, discontinuous, unpredictably varying in computational expense, and include continuous, categorical, and/or integer variables. Further, function evaluations can fail for a variety of reasons including numerical difficulties or hardware failures. Additionally, not all hyperparameter value combinations are compatible, which creates so called hidden constraints. Robust and efficient optimization algorithms are needed for hyperparameter tuning. In this paper we present an automated parallel derivative-free optimization framework called Autotune , which combines a number of specialized sampling and search methods that are very effective in tuning machine learning models despite these challenges. Autotune provides significantly improved models over using default hyperparameter settings with minimal user interaction on real-world applications. Given the inherent expense of training numerous candidate models, we demonstrate the effectiveness of Autotune's search methods and the efficient distributed and parallel paradigms for training and tuning models, and also discuss the resource trade-offs associated with the ability to both distribute the training process and parallelize the tuning process.

Subfields: Biochemistry, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biochemistry

1. Marc-André Zöller
2. Marco F. Huber

In Computational Mechanics

1. Marc-André Zöller
2. Shuicheng Yan
3. Chao Du
4. Min Lin
5. Tianyu Pang
6. Marco F. Huber
7. Bingyi Kang
8. Takuya Akiba

In Computer Vision and Pattern Recognition

1. Shuicheng Yan
2. Yan Huang
3. Jiashi Feng
4. Bingyi Kang
5. Yi Yang
6. Ming-Hsuan Yang
7. Dacheng Tao
8. Luc Van Gool

Citations in the corpus

• Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “Various open-source designs for AutoML services have been proposed (e.g., Sparks et al., 2015; Chan, 2017; Swearingen et al., 2017; Koch et al., 2018) but also several commercial solutions exist (e.g., Golovin et al., 2017; Clouder, 2018; H2O.ai, 2018).”
• Survey on Automated Machine Learning (2019, methodology): “Various open-source designs for AutoML services have been proposed, e.g., (Sparks et al., 2015; Chan, 2017; Swearingen et al., 2017; Koch et al., 2018), but also several commercial solutions exist, e.g., (Golovin et al., 2017; Clouder, 2018; H2O.ai, 2018).”
• Bag of Tricks for Training Data Extraction from Language Models (2023, methodology): “We use a versatile architecture auto-tuning method (Akiba et al., 2019), which incorporates efficient search and pruning strategies, to determine the optimized hyperparameters following advanced frameworks (Snoek et al., 2012; Koch et al., 2018; Akiba et al., 2019).”
• Optuna: A Next-generation Hyperparameter Optimization Framework (2019, methodology): “Compatibility with modular programming SMAC [3] define-and-run GPyOpt define-and-run Spearmint [2] define-and-run Hyperopt [1] define-and-run Autotune [4] define-and-run Vizier [5] define-and-run Katib define-and-run Tune [7] define-and-run Optuna (thiswork) define-by-run def create rf(trial): return RandomF”