Massively Parallel Hyperparameter Tuning

By Liam Li, Kevin G. Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, B. Recht, Ameet Talwalkar. 2018

193 citations recorded by the source corpus.

Subfields: Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Neural Architecture Search & AutoML

1. Frank Hutter
2. Aaron Klein
3. Stefan Falkner
4. Ameet Talwalkar
5. Liam Li
6. Zhenguo Li
7. Yujun Li
8. Zhihua Zhang

Citations in the corpus

• Quantifying the Carbon Emissions of Machine Learning (2019, background): “However, it has been shown that random search (and others) not only is a straightforward replacement but also has potential to significantly accelerate hyperparameter search [20, 21, 22], consequently reducing carbon emissions.”
• BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “We note that SH has also been parallelized in (so far un-published) independent work (Li et al., 2018).”
• Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “The computational resources are automatically allocated by ASHA, and thus there are no extra hyper-parameters for users compared with HB.”
• Random Search and Reproducibility for Neural Architecture Search (2019, methodology): “Specifically, we evaluate a general hyperparameter optimization method combining random search with early-stopping [28] on two standard NAS benchmarks (CIFAR-10 and PTB).”
• Tune: A Research Platform for Distributed Model Selection and Training (2018, methodology): “Lines of code FIFO (trivial scheduler) 10 Asynchronous HyperBand (Li et al. (2018)) 78 HyperBand (Li et al. (2016) coordinate with each other using Ray to e.g., perform SGD, or use collective communication primitives provided by libraries such as torch.distributed and Nvidia NCCL.”
• Use What You Have: Video Retrieval Using Representations From Collaborative Experts (2020, methodology): “To determine zero-padding, margin and VLAD clusters for DiDeMo, MSVD and LSMDC further Asynchronous Hyperband searches were conducted, each with a budget 1Note: referred to by [22] as the JMET-JMDV split 2https://sites.google.com/site/describingmovies/lsmdc-2017 of 500 sampled configurations.”
• Limitations of Weak Labels for Embedding and Tagging (2020, unknown): “The architecture of the model (number of convolution layers, dropout, embedding size, number of fully connected layers) has been optimized on the end-to-end classifier with the Asynchronous Successive Halving Algorithm (the asynchronous version of Hyperband) [30], using Orion(3).”
• An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization (2020, methodology): “In the cases with large deviations, the advantages of MSS and AMSS over SH and ASHA are more obviously.”
• Optuna: A Next-generation Hyperparameter Optimization Framework (2019, methodology): “Op-tuna features a variant of Asynchronous Successive Halving algorithm [19] , a recently developed state of the art method that scales linearly with the number of workers in distributed environment.”
• MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages (2022, methodology): “For each model, we performed 128 trials of hyperparameter tuning using the Tree of Parzen Estimators algorithm and Asynchronous Successive Halving Algorithm (ASHA) (Li et al., 2018a) for scheduling, which are both part of the hyperopt library (Bergstra et al., 2013) integrated into the ray[tune] lib”
• Unreproducible Research is Reproducible (2019, methodology): “ASHA required training 30k models for 15 epochs, 7k+ models for 30 epochs and 1k+ models for 60 epochs.”
• Knowledge Transfer Graph for Deep Collaborative Learning (2019, methodology): “We used the Asynchronous Successive Halving Algorithm (ASHA) [16] as the hyperparameter optimization method.”