A method for solving the convex programming problem with convergence rate O(1/k^2)

By Yurii E. Nesterov. 1983

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “Improving Training Speed and Efficiency There is a large body of work on accelerating model training, traditionally accomplished via improved optimizers (Nesterov, 1983; Kingma & Ba, 2015).”
Don't Decay the Learning Rate, Increase the Batch Size (2017, methodology): “In ﬁgure 3b, we present the test set accuracy, when training with Nesterov momentum (Nesterov, 1983) and momentum parameter 0.9, observing three near-identical curves.”
On Empirical Comparisons of Optimizers for Deep Learning (2019, background): “SGD is due to Robbins and Monro (1951), M OMENTUM to Polyak (1964), N ESTEROV to Nesterov (1983), RMSP ROP to Tieleman and Hinton (2012), RMS TEROV is our own, A DAM to Kingma and Ba (2015), and NA DAM to Dozat (2016).”
Bag of Tricks for Image Classification with Convolutional Neural Networks (2018, background): “Nesterov Accelerated Gradient (NAG) descent [20] is used for training.”
SGDR: Stochastic Gradient Descent with Warm Restarts (2016, background): “Recently, O’Donoghue & Candes (2012) noted that the iterates of accelerated gradient schemes proposed by Nesterov (1983; 2013) exhibit a periodic behavior if momentum is overused.”
Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “, stochastic gradient descent (Nesterov, 1983), Adam (Kingma and Ba, 2014), RMSProp (Hinton et al.”
Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, methodology): “• We use Nesterov’s Accelerated Gradient (Nesterov, 1983) with a momentum parameter of 0.”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “Our ﬁrst assumption is that we use an optimizer which involves gradient moments (Nesterov, 1983; Tiele-man and Hinton, 2012; Kingma and Ba, 2015; Loshchilov and Hutter, 2019; Shazeer and Stern, 2018).”
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, background): “D N-LAMB: N ESTEROV M OMENTUM FOR LAMB Sutskever et al. (2013) report that Nesterov’s accelerated gradient (NAG) proposed by Nesterov (1983) is conceptually and empirically better than the regular momentum method for convex, non-stochastic objectives.”
Efficient Neural Architecture Search via Parameter Sharing (2018, methodology): “The shared parameters ω are trained with Nesterov momentum (Nesterov, 1983), where the learning rate follows the cosine schedule with l max = 0 .”
Optimization Methods for Large-Scale Machine Learning (2016, methodology): “1), but with its own unique properties, is the accelerated gradient method proposed by Nesterov [106].”
GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium (2017, background): “Previous to NAGD there was Polyak’s Heavy Ball method [44].”
On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization (2018, methodology): “…analyses, to resemble a combination of two well-known tools in the ﬁeld of optimization: momentum , which led to provable acceleration bounds (Nesterov, 1983); and adaptive regularization , a more recent technique proven to accelerate by Duchi et al. (2011) in their proposal of the AdaGrad…”
Federated Evaluation of On-device Personalization (2019, methodology): “The server update is achieved via the Momentum optimizer, using Nesterov accelerated gradient (Nesterov, 1983; Sutskever et al., 2013), a momentum hyperparameter of 0.9, and a server learning rate of 1.0.”
Federated Learning Of Out-Of-Vocabulary Words (2019, methodology): “FL M L has the same model architecture as FLSGD L and further applies adaptive gradient clipping, combined with Nesterov accelerated gradient (Nesterov, 1983) and a momentum hyper-parameter of 0.”
Federated Learning for Mobile Keyboard Prediction (2018, methodology): “The server update in Equation 3 is achieved via the Momentum optimizer, using Nesterov accelerated gradient [29], a momentum hyperparameter of 0.”
Stochastic Gradient Push for Distributed Deep Learning (2018, methodology): “While our theory focuses on analyzing SGP, which combines PUSHSUM with SGD, our experiments illustrate that PUSHSUM can similarly be efficiently combined with other optimizers like Nesterov’s accelerated gradient method (Nesterov, 1983) and Adam (Kingma & Ba, 2015).”
Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimization (2018, methodology): “[20] and the best known algorithms at the time inspired Nesterov’s accelerated gradient descent algorithm [21].”