Fast incremental method for smooth nonconvex optimization

By Sashank J. Reddi, Suvrit Sra, Barnab'as P'oczos, Alex Smola. 2016

71 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Mechanics, Numerical Analysis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Numerical Analysis

Citations in the corpus

AdaGrad stepsizes: Sharp convergence over nonconvex landscapes, from any initialization (2018, methodology): “In the noisy setting, faster rates than O ( 1/ √ N ) are also possible using accelerated SGD methods (Ghadimi and Lan, 2016; AllenZhu and Yang, 2016; Reddi et al., 2016; Allen-Zhu, 2017; Xu et al., 2018; Zhou et al., 2018; Fang et al., 2018).”
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning (2019, methodology): “These exactly match the rates of SAGA (Defazio et al., 2014; Reddi et al., 2016c).”
Solving Weakly-Convex-Weakly-Concave Saddle-Point Problems as Successive Strongly Monotone Variational Inequalities (2018, background): “Instead of considering a min-max problem, most of them contribute to the non-convex minimization problem [29, 70, 30, 63, 61, 44, 1, 2, 25, 24, 72, 24].”
Nonconvex Variance Reduced Optimization with Arbitrary Sampling (2018, methodology): “VR methods were recently extended to work with (non-strongly) convex losses [13], and more recently also to non-convex losses [29, 28, 2, 23], in all cases leading to best current rates for (1) in a given function class.”
Stochastic Frank-Wolfe methods for nonconvex optimization (2016, background): “Similar (but not same) benefits have also been reported for nonconvex Svrg and Saga over gradient descent (Reddi et al., 2016a;b).”
Global Convergence and Variance-Reduced Optimization for a Class of Nonconvex-Nonconcave Minimax Problems (2020, methodology): “…a prominent role in improving the convergence over stochastic or batch algorithms for both convex and nonconvex minimization problems, which have been extensively studied in the past few years; see, e.g., [Johnson and Zhang, 2013, Reddi et al., 2016a,b, Xiao and Zhang, 2014], just to name a few.”