How To Make the Gradients Small Stochastically

By Zeyuan Allen-Zhu. 2018

17 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Byzantine Stochastic Gradient Descent (2018, positive): “eable with the criterion of 2 kx x k2 "). In some applications, one may also consider nding a point x with kr f (x )k ". This task is harder and requires additional work on top of SGD [2] (even without Byzantine machines). Our results of this paper generalize to that setting, but for si”
The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure (2019, methodology): “Allen-Zhu (2018) present an alternative approach towards minimizing the gradient norm with access to stochastic gradients.”
Asymmetric Valleys: Beyond Sharp and Flat Local Minima (2019, background): “For example, it is shown that they could escape saddle points or sharp local minima under reasonable assumptions (Ge et al., 2015; Jin et al., 2017; 2018a;b; Xu et al., 2018; Allen-Zhu, 2018a;b; Allen-Zhu & Li, 2018; Kleinberg et al., 2018).”
Complexity of finding near-stationary points of convex functions stochastically (2018, methodology): “In this short note, we show that the gradual regularization technique of Allen-Zhu [1], along with averaging of the iterates, improves the complexity to O ( ε − 2 ) in complete analogy to the smooth setting.”
Stochastic model-based minimization of weakly convex functions (2018, methodology): “By following a completely diﬀerent technique, introduced by Allen-Zhu [5] for smooth stochastic minimization, this complexity can be even further improved to by running logarithmically many rounds of the subgradient method on quadratically regularized problems.”