Forward and Reverse Gradient-Based Hyperparameter Optimization

By Luca Franceschi, Michele Donini, Paolo Frasconi, Massimiliano Pontil. 2017

477 citations recorded by the source corpus.

We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm such as stochastic gradient descent. These procedures mirror two methods of computing gradients for recurrent neural networks and have different trade-offs in terms of running time and space requirements. Our formulation of the reverse-mode procedure is linked to previous work by Maclaurin et al. [2015] but does not require reversible dynamics. The forward-mode procedure is suitable for real-time hyperparameter updates, which may significantly speed up hyperparameter optimization on large datasets. We present experiments on data cleaning and on learning task interactions. We also present one large-scale experiment where the use of previous gradient-based methods would be prohibitive.

Subfields: Artificial Intelligence, Optimization & Training, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Yoshua Bengio
  3. Quoc V. Le
  4. Mohit Bansal
  5. Dacheng Tao
  6. Pieter Abbeel
  7. Jianfeng Gao
  8. Percy Liang

In Optimization & Training

  1. Sergey Levine
  2. Qiang Yang
  3. Chelsea Finn
  4. Massimiliano Pontil
  5. Amos J. Storkey
  6. Hado van Hasselt
  7. David Silver
  8. Li Yang

In Neural Architecture Search & AutoML

  1. Frank Hutter
  2. Arber Zela
  3. Marius Thomas Lindauer
  4. Thomas Elsken
  5. Li Yang
  6. Abdallah Shami
  7. Thomas Brox
  8. Tonmoy Saikia

Citations in the corpus