The Physical Systems Behind Optimization Algorithms
By Lin F. Yang, Raman Arora, Vladimir Braverman, Tuo Zhao. 2016
19 citations recorded by the source corpus.
We use differential equations based approaches to provide some {\it \textbf{physics}} insights into analyzing the dynamics of popular optimization algorithms in machine learning. In particular, we study gradient descent, proximal gradient descent, coordinate gradient descent, proximal coordinate gradient, and Newton's methods as well as their Nesterov's accelerated variants in a unified framework motivated by a natural connection of optimization algorithms to physical systems. Our analysis is applicable to more general algorithms and optimization problems {\it \textbf{beyond}} convexity and strong convexity, e.g. Polyak-Łojasiewicz and error bound conditions (possibly nonconvex).
Subfields: Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Optimization & Training
- Guodong Zhang
- Roger Baker Grosse
- James Martens
- George E. Dahl
- Christopher J. Shallue
- Laurence Aitchison
- Philipp Hennig
- Zachary Nado
Citations in the corpus
- Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, methodology): “Following the existing treatment of the deterministic version of the algorithm [Chiang, 1974, Qian, 1999, Yang et al., 2018, Goh, 2017], we divide our analysis two cases: overdamping and underdamping.”