Introductory Lectures on Convex Optimization - A Basic Course

By Yurii E. Nesterov. 2014

6,525 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Weight Standardization (2019, background): “Speciﬁ-cally, [37] shows that BN reduces the Lipschitz constants of the loss function, and makes the gradients more Lips-chitz, too, i.e. , the loss will have a better β -smoothness [29].”
SGDR: Stochastic Gradient Descent with Warm Restarts (2016, background): “Recently, O’Donoghue & Candes (2012) noted that the iterates of accelerated gradient schemes proposed by Nesterov (1983; 2013) exhibit a periodic behavior if momentum is overused.”
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour (2017, methodology): “We use Nesterov momentum [28] with m of 0.9 following [12] but note that standard momentum as was used in [16] is equally effective.”
Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity (2021, background): “According to Theorem 2.1.5 in Nesterov [2013], if f i is convex and L i smooth, then the following two conditions are equivalent: If in the above two condition we select y = x ∗ and take the expectations with respect to i , then we obtain the following two equivalent conditions: Note that in the…”
A Tight and Unified Analysis of Extragradient for a Whole Spectrum of Differentiable Games (2019, methodology): “Interestingly, the examples we end up using have a spectrum similar to the one of the matrix Nesterov uses in the proofs of his lower bounds in Nesterov (2014).”
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, background): “However, the practical efficiency of these methods is adversely affected by two challenges: (i) their performance relies heavily on the choice of the step-size (“learning rate”) [9, 64] and (ii) SGD can converge slowly compared to methods that compute the full gradient (over all training examples) i”
Frank-Wolfe Algorithms for Saddle Point Problems (2016, methodology): “The standard approaches to solve smooth constrained saddle point problems are projection-type methods (surveyed in Xiu and Zhang (2003)), with in particular variations of Korpelevich’s extragradient method (Korpelevich, 1976), such as (Nesterov, 2007) which was used to solve the structured predictio”
Barrier Frank-Wolfe for Marginal Inference (2015, neutral): “2 in Nesterov (2004)).”
Optimization Methods for Large-Scale Machine Learning (2016, background): “erate averaging methods 20 do not neatly belong in the category of second-order methods, even though we place them there, and one could argue that gradient methods with momentum [120] or acceleration [106,107] do belong in this category, even though we discuss them separately in x7. Nevertheless, Fi”
A Variational Inequality Perspective on Generative Adversarial Networks (2018, neutral): “…an (extrapolated) point different from the current point from which the update is performed, stabilizing the dynamics: Compute extrapolated point: Perform update step: Note that, even in the unconstrained case , this method is intrinsically different from Nesterov’s momentum 6 (Nesterov, 1983, Eq.”
Natural Compression for Distributed Deep Learning (2019, background): “, 2017), importance sampling (Horváth and Richtárik, 2019), momentum (Nesterov, 2013), or variance reduction (Johnson and Zhang, 2013)), is the cost of communication of the typically dense gradient vector gi(x), and in a parameter-sever implementation with a master node, also the cost of broadcastin”