A method for unconstrained convex minimization problem with the rate of convergence o(1/k^2)

By Yurii E. Nesterov. 1983

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Measuring the Effects of Data Parallelism on Neural Network Training (2018, background): “, 2013), Nesterov momentum (Nesterov, 1983; Sutskever et al., 2013), RMSProp (Hinton et al.”
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, positive): “The word-piece and word-level GCNN models are trained with Nesterov accelerated gradient descent (Nesterov, 1983) on 8 GPUs for 22 epochs with a step-wise learning rate schedule starting from 1 and decreasing by a factor of 5 when the loss is on the plateau.”
High-Performance Large-Scale Image Recognition Without Normalization (2021, methodology): “We train using the categorical softmax cross-entropy loss with label smoothing of 0.1 (Szegedy et al., 2016b), and optimize our networks using stochastic gradient descent (Robbins & Monro, 1951) with Nesterov’s momentum (Nesterov, 1983; Sutskever et al., 2013), using a momentum coefficient of 0.9.”
BinaryConnect: Training Deep Neural Networks with binary weights during propagations (2015, methodology): “Last but not least, we scale the weights learning rates respectively with the weights initialization coefficients from [25] when optimizing with ADAM, and with the squares of those coefficients when optimizing with SGD or Nesterov momentum [28].”
Advances in optimizing recurrent networks (2012, neutral): “We select hyperparameters, such as the number of hidden unitsnh, regularization coefficientsλL1, the choice of non-linearity function, or the momentum scheduleµt, learning rateǫt, number of leaky unitsnleaky or leaky factorsα according to log-likelihood on a validation set and we report the final…”
Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Networks (2018, methodology): “For optimization, we used SGD with momentum (Qian, 1999; Nesterov, 1983).”
Practical tradeoffs between memory, compute, and performance in learned optimizers (2022, background): “The parameters may be, for example, neural network weights, whereas the optimizer state includes quantities such as the accumulated momentum values in momentum-accelerated optimizers (Polyak, 1964; Nesterov, 1983).”
ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning (2020, methodology): “Due to their simplicity and effectiveness, first order optimization methods [20, 29, 34, 41, 49, 71] have become the de-facto algorithms used in deep learning.”
Adversarial Neural Machine Translation (2017, methodology): “For the training of NMT modelG, similar as what is commonly done in previous works (Shen et al., 2016; Tu et al., 2016a), we warm start G from a well-trained RNNSearch model, and optimize it using vanilla SGD with mini-batch size 80 for En→Fr translation and 32 for De→En translation.”
Parallelizing Stochastic Gradient Descent for Least Squares Regression: Mini-batching, Averaging, and Model Misspecification (2016, methodology): “Geometric Rates on initial error: For ofﬂine optimization with strongly convex objectives, gradient descent (Cauchy, 1847) and fast gradient methods (Polyak, 1964; Nesterov, 1983) indicate linear convergence.”