A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights

By Weijie Su, Stephen P. Boyd, E.J. Candes. 2014

1,385 citations recorded by the source corpus.

We derive a second-order ordinary differential equation (ODE), which is the limit of Nesterov's accelerated gradient method. This ODE exhibits approximate equivalence to Nesterov's scheme and thus can serve as a tool for analysis. We show that the continuous time ODE allows for a better understanding of Nesterov's scheme. As a byproduct, we obtain a family of schemes with similar convergence rates. The ODE interpretation also suggests restarting Nesterov's scheme leading to an algorithm, which can be rigorously proven to converge at a linear rate whenever the objective is strongly convex.

Subfields: Artificial Intelligence, Computational Mechanics, Numerical Analysis, Optimization & Training, Mathematical Optimization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Michael I. Jordan
  2. Yoshua Bengio
  3. Ruslan Salakhutdinov
  4. Sergey Levine
  5. Percy Liang
  6. Quoc V. Le
  7. Mohit Bansal
  8. Jianfeng Gao

In Computational Mechanics

  1. Michael I. Jordan
  2. Simon Shaolei Du
  3. Francis R. Bach
  4. Sanjeev Arora
  5. Praneeth Netrapalli
  6. Chi Jin
  7. Peter L. Bartlett
  8. Zeyuan Allen-Zhu

In Numerical Analysis

  1. Michael I. Jordan
  2. Hédy Attouch
  3. Ernö Robert Csetnek
  4. B. Recht
  5. James Lucas
  6. Roger Baker Grosse
  7. Alexandre Cabot
  8. Richard S. Zemel

In Optimization & Training

  1. Michael I. Jordan
  2. Peter L. Bartlett
  3. Simon Lacoste-Julien
  4. Simon Shaolei Du
  5. Roger Baker Grosse
  6. James Lucas
  7. Peter Richtárik
  8. Nicolas Loizou

In Mathematical Optimization

  1. Michael I. Jordan
  2. Ashia C. Wilson
  3. Chi Jin
  4. Yura Malitsky
  5. Hédy Attouch
  6. Roger Baker Grosse
  7. James Lucas
  8. Richard S. Zemel

Citations in the corpus