A System for Massively Parallel Hyperparameter Tuning
By Liam Li, Kevin G. Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-tzur, Moritz Hardt, B. Recht, Ameet Talwalkar. 2018
521 citations recorded by the source corpus.
Modern learning models are characterized by large hyperparameter spaces and long training times. These properties, coupled with the rise of parallel computing and the growing demand to productionize machine learning workloads, motivate the need to develop mature hyperparameter optimization functionality in distributed computing settings. We address this challenge by first introducing a simple and robust hyperparameter optimization algorithm called ASHA, which exploits parallelism and aggressive early-stopping to tackle large-scale hyperparameter optimization problems. Our extensive empirical results show that ASHA outperforms existing state-of-the-art hyperparameter optimization methods; scales linearly with the number of workers in distributed settings; and is suitable for massive parallelism, as demonstrated on a task with 500 workers. We then describe several design decisions we encountered, along with our associated solutions, when integrating ASHA in Determined AI's end-to-end production-quality machine learning system that offers hyperparameter tuning as a service.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Management Science and Operations Research, Neural Architecture Search & AutoML, ML Systems & Hardware
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Quoc V. Le
- Yoshua Bengio
- Percy Liang
- Mohit Bansal
- Sergey Levine
- Weizhu Chen
- Luke S. Zettlemoyer
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jitendra Malik
- Xiaogang Wang
- Ming-Hsuan Yang
- Yi Yang
- Andrew Zisserman
In Management Science and Operations Research
- Liam Li
- Ameet Talwalkar
- Marc-André Zöller
- Marco F. Huber
- Afshin Rostamizadeh
- Frank Hutter
- B. Recht
- Moritz Hardt
In Neural Architecture Search & AutoML
- Liam Li
- Ameet Talwalkar
- B. Recht
- Moritz Hardt
- Frank Hutter
- Marc-André Zöller
- Afshin Rostamizadeh
- Kevin G. Jamieson
In ML Systems & Hardware
- Marc-André Zöller
- Pratik Chaudhari
- Ameet Talwalkar
- Christian Theobalt
- Taghi M. Khoshgoftaar
- Marco F. Huber
- Tong Yu
- Hong Zhu
Citations in the corpus
- Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “The computational resources are automatically allocated by ASHA, and thus there are no extra hyper-parameters for users compared with HB.”
- Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “By evaluating multiple problem instances in parallel, long running instances can be discarded early (Weisz et al., 2018; Li et al., 2020).”
- Model Zoo: A Growing Brain That Learns Continually (2021, methodology): “The Async Successive Halving Algorithm (ASHA) scheduler (Li et al., 2018) was used to prune hyper-parameter choices with the search space determined by Nevergrad (Rapin and Teytaud, 2018).”
- An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization (2020, methodology): “In the cases with large deviations, the advantages of MSS and AMSS over SH and ASHA are more obviously.”
- FLAML: A Fast and Lightweight AutoML Library (2019, unknown): “2019; Li et al., 2020).”
- Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (2022, background): “Hyperparameter Tuning Many have sought to speed up HP tuning beyond the simple grid or random search, such as via Bayesian optimization [34, 35] or multi-arm bandits [15, 18].”
- Text Data Augmentation for Deep Learning (2021, background): “Hyperparam-eter optimization is one of the active areas of Deep Learning research [115–117].”
- Advances in Neural Rendering (2021, methodology): “Ray tune [LLN ∗ 18] has implementations of algorithms like ASHA [LJR ∗ 20] and Hyperband [LJRT18], which can dynamically assign computational and time budgets to experiments for a faster hyperparameter search.”
- Massively Parallel Hyperparameter Tuning (2018, methodology): “SHA [Jamieson and Talwalkar, 2015] and Hyperband [Li et al., 2018] are empirically effective adaptive evaluation techniques that employ principled ‘early-stopping.’”
- The Benchmark Lottery (2021, unknown): “Spending enough time and compute to precisely tune hyper-parameters of the model or the training process can improve the results a great deal [Li et al., 2018, Bello et al., 2021, Steiner et al., 2021].”