ADADELTA: An Adaptive Learning Rate Method
By Matthew D. Zeiler. 2012
6,871 citations recorded by the source corpus.
We present a novel per-dimension learning rate method for gradient descent called ADADELTA. The method dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent. The method requires no manual tuning of a learning rate and appears robust to noisy gradient information, different model architecture choices, various data modalities and selection of hyperparameters. We show promising results compared to other methods on the MNIST digit classification task using a single machine and on a large scale voice dataset in a distributed cluster environment.
Subfields: Artificial Intelligence, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Quoc V. Le
- Percy Liang
- Luke S. Zettlemoyer
- Graham Neubig
- Kyunghyun Cho
In Optimization & Training
- Yoshua Bengio
- Frank Hutter
- Simon Lacoste-Julien
- Quoc V. Le
- Peter Richtárik
- Roger Baker Grosse
- Philipp Hennig
- Nicolas Loizou
Citations in the corpus
- Adam: A Method for Stochastic Optimization (2014, methodology): “Other stochastic optimization methods include vSGD (Schaul et al., 2012), AdaDelta (Zeiler, 2012) and the natural Newton method from Roux & Fitzgibbon (2010), all setting stepsizes by estimating curvature from first-order information.”
- Neural Machine Translation by Jointly Learning to Align and Translate (2014, methodology): “We use a minibatch stochastic gradient descent (SGD) algorithm together with Adadelta (Zeiler, 2012) to train each model.”
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, methodology): “95 (Zeiler, 2012).”
- Deep Contextualized Word Representations (2018, neutral): “95 (Zeiler, 2012).”
- Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “We train the model with the Adadelta optimizer (Zeiler, 2012) with a batch size 60 for Triv-iaQA and 45 for SQuAD.”
- A large annotated corpus for learning natural language inference (2015, methodology): “We additionally transfer the accumulators that are used by AdaDelta to set the learning rates.”
- Learning to learn by gradient descent by gradient descent (2016, background): “We compare our trained optimizers with standard optimizers used in Deep Learning: SGD, RMSprop, ADAM, Adadelta, Adagrad, and Rprop.”
- Adafactor: Adaptive Learning Rates with Sublinear Memory Cost (2018, methodology): “…is divided by the componentwise square root of a vector summarizing the history of squared gradients, usually obtained through summation as in Adagrad (Duchi et al., 2011) or exponential averaging as in RMSProp (Tieleman & Hinton, 2012), Adam (Kingma & Ba, 2015), and Adadelta (Zeiler, 2012).”
- MADE: Masked Autoencoder for Distribution Estimation (2015, methodology): “The experiments were run with networks of 500 units per hidden layer, using the adadelta learning update (Zeiler, 2012) with a decay of 0.95.”
- Towards Exploiting Background Knowledge for Building Conversation Systems (2018, methodology): “Again following the original paper, we trained BiDAF using AdaDelta (Zeiler, 2012) optimizer with an initial learning rate of 0.5 on a minibatch of size 32.”
- Asynchronous Methods for Deep Reinforcement Learning (2016, methodology): “ADADELTA [Zeiler, 2012], RMSProp [Tieleman and Hinton, 2012] and ADAM [Kingma and Ba, 2014], there is no consensus as to which method is the best.”
- Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback (2018, neutral): “The SGD optimizer used is ADADELTA (Zeiler, 2012).”
- Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “Stochastic gradient descent (SGD) and its variants [22, 91, 40, 84, 74, 36, 19] are the preferred optimization methods in modern machine learning.”