ADADELTA: An Adaptive Learning Rate Method

By Matthew D. Zeiler. 2012

6,871 citations recorded by the source corpus.

We present a novel per-dimension learning rate method for gradient descent called ADADELTA. The method dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent. The method requires no manual tuning of a learning rate and appears robust to noisy gradient information, different model architecture choices, various data modalities and selection of hyperparameters. We show promising results compared to other methods on the MNIST digit classification task using a single machine and on a large scale voice dataset in a distributed cluster environment.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Yoshua Bengio
  4. Quoc V. Le
  5. Percy Liang
  6. Luke S. Zettlemoyer
  7. Graham Neubig
  8. Kyunghyun Cho

In Optimization & Training

  1. Yoshua Bengio
  2. Frank Hutter
  3. Simon Lacoste-Julien
  4. Quoc V. Le
  5. Peter Richtárik
  6. Roger Baker Grosse
  7. Philipp Hennig
  8. Nicolas Loizou

Citations in the corpus