Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

By John C. Duchi, Elad Hazan, Yoram Singer. 2011

11,425 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Management Science and Operations Research, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Optimization & Training

Citations in the corpus

Adam: A Method for Stochastic Optimization (2014, methodology): “Our Theorem 4.1 implies when the data features are sparse and bounded gradients, the summation term can be much smaller than its upper bound √ T , in particular if the class of function and data features are in the form of section 1.2 in (Duchi et al., 2011).”
Structured Attention Networks (2017, background): “Additional training details include: batch size of 32 ; training for 100 epochs with Adagrad (Duchi et al., 2011) where the global learning rate is 0 .”
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “This technique could similarly be applied to Adagrad (Duchi et al., 2010).”
Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, methodology): “We perform stochastic gradient descent with AdaGrad (Duchi et al., 2011), subsampling CONJUNCTION and NEXT by a factor of 4 and 6 respectively (chosen using held-out accuracy averaged over all three tasks on held out data after training on 1M examples).”
A Decomposable Attention Model for Natural Language Inference (2016, methodology): “Each hyperparameter setting was run on a single machine with 10 asynchronous gradient-update threads, using Adagrad (Duchi et al., 2011) for optimization with the default initial accumulator value of 0.”
GloVe: Global Vectors for Word Representation (2014, methodology): “For all our experiments, we set x max = 100, α = 3 / 4, and train the model using AdaGrad (Duchi et al., 2011), stochastically sampling non-zero elements from X , with initial learning rate of 0.05.”
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank (2013, methodology): “For all models, we use the dev set and cross-validate over regularization of the weights, word vector size as well as learning rate and minibatch size for AdaGrad.”
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015, methodology): “, 2013) and Adagrad (Duchi et al., 2011) have been used to achieve state of the art performance.”
Auto-Encoding Variational Bayes (2013, positive): “Derivatives ∇ θ , φ (cid:101) L ( θ ; X M ) can be taken, and the resulting gradients can be used in conjunction with stochastic optimization methods such as SGD or Adagrad [DHS10].”
Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, methodology): “ADAGrad indicates the AdaGrad algorithm, with the trailing number indicating the initial step size.”
ADADELTA: An Adaptive Learning Rate Method (2012, methodology): “The idea presented in this paper was derived from ADA-GRAD [3] in order to improve upon the two main drawbacks of the method: 1) the continual decay of learning rates throughout training, and 2) the need for a manually selected global learning rate.”
No more pesky learning rates (2012, methodology): “Finally, derived from a worst-case analysis, (Duchi et al., 2010) propose an approach called ‘ AdaGrad ’, where the learning rate takes the form The main practical drawback for all of these approaches is that they retain one or more sensitive hyper-parameters, which must be tuned to obtain…”
Deep Generative Stochastic Networks Trainable by Backprop (2013, methodology): “For example, one could use a convolutional architecture with max-pooling for parametric parsimony and computational efficiency, Adagrad (Duchi et al., 2011) for quick gradient descent, and dropout (Hinton et al.”
Learning to Optimize Neural Nets (2017, methodology): “We use it to learn an optimization algorithm for training shallow neural nets and show that it outperforms popular hand-engineered optimization algorithms like ADAM (Kingma & Ba, 2014), AdaGrad (Duchi et al., 2011) and RMSprop (Tieleman & Hinton, 2012) and an optimization algorithm learned using the”
Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, neutral): “(2) Moment methods and AdaGrad training (Duchi et al., 2011) speed up the convergence speed but gave similar results as plain SGD.”
A Neural Network for Factoid Question Answering over Paragraphs (2014, methodology): “The parameters θ = (Wr∈R,Wv,We, b), where R represents all dependency relations in the data, are optimized using AdaGrad(Duchi et al., 2011).”
Semantic Parsing on Freebase from Question-Answer Pairs (2013, methodology): “We optimize the objective by initializing θ0 to 0 and applying AdaGrad (stochastic gradient ascent with per-feature adaptive step size control) (Duchi et al., 2010), so that θt+1 is set based on taking a stochastic approximation of ∂O(θ;θt)∂θ ∣∣ θ=θt .”
Efficient Estimation of Word Representations in Vector Space (2013, methodology): “For this parallel training, we use mini-batch asynchronous gradient descent with an adaptive learning rate procedure called Adagrad [7].”