A System for Massively Parallel Hyperparameter Tuning

By Liam Li, Kevin G. Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-tzur, Moritz Hardt, B. Recht, Ameet Talwalkar. 2018

521 citations recorded by the source corpus.

Modern learning models are characterized by large hyperparameter spaces and long training times. These properties, coupled with the rise of parallel computing and the growing demand to productionize machine learning workloads, motivate the need to develop mature hyperparameter optimization functionality in distributed computing settings. We address this challenge by first introducing a simple and robust hyperparameter optimization algorithm called ASHA, which exploits parallelism and aggressive early-stopping to tackle large-scale hyperparameter optimization problems. Our extensive empirical results show that ASHA outperforms existing state-of-the-art hyperparameter optimization methods; scales linearly with the number of workers in distributed settings; and is suitable for massive parallelism, as demonstrated on a task with 500 workers. We then describe several design decisions we encountered, along with our associated solutions, when integrating ASHA in Determined AI's end-to-end production-quality machine learning system that offers hyperparameter tuning as a service.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Management Science and Operations Research, Neural Architecture Search & AutoML, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Quoc V. Le
  3. Yoshua Bengio
  4. Percy Liang
  5. Mohit Bansal
  6. Sergey Levine
  7. Weizhu Chen
  8. Luke S. Zettlemoyer

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Jitendra Malik
  5. Xiaogang Wang
  6. Ming-Hsuan Yang
  7. Yi Yang
  8. Andrew Zisserman

In Management Science and Operations Research

  1. Liam Li
  2. Ameet Talwalkar
  3. Marc-André Zöller
  4. Marco F. Huber
  5. Afshin Rostamizadeh
  6. Frank Hutter
  7. B. Recht
  8. Moritz Hardt

In Neural Architecture Search & AutoML

  1. Liam Li
  2. Ameet Talwalkar
  3. B. Recht
  4. Moritz Hardt
  5. Frank Hutter
  6. Marc-André Zöller
  7. Afshin Rostamizadeh
  8. Kevin G. Jamieson

In ML Systems & Hardware

  1. Marc-André Zöller
  2. Pratik Chaudhari
  3. Ameet Talwalkar
  4. Christian Theobalt
  5. Taghi M. Khoshgoftaar
  6. Marco F. Huber
  7. Tong Yu
  8. Hong Zhu

Citations in the corpus