Stochastic Mirror Descent in Variationally Coherent Optimization Problems

By Zhengyuan Zhou, Panayotis Mertikopoulos, N. Bambos, Stephen P. Boyd, Peter W. Glynn. 2017

88 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity (2021, methodology): “In addition in Zhou et al. [2017], the strongly coherent optimization problems have been studied, which are special cases of the quasi-strongly monotone VI.”
Stochastic Mirror Descent on Overparameterized Nonlinear Models (2019, background): “Mirror descent, ﬁrst introduced by Nemirovski & Yudin (1983), is one of the most widely used families of algorithms for optimization (Beck & Teboulle, 2003; Cesa-Bianchi et al., 2012; Zhou et al., 2017), which includes the popular gradient descent as a special case.”
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels (2017, background): “Under standard assumptions, Theorem 1 shows that the algo-rithm stabilizes and converges to a stationary point (convergence to global/local minima cannot be guaranteed unless in specially structured non-convex objectives (Chen et al., 2018; Zhou et al., 2017b;a)).”
Efficient Methods for Structured Nonconvex-Nonconcave Min-Max Optimization (2020, background): “The condition that we impose on F that enables our convergence results differs from the strong assumption that a solution to ( MVI ) exists [32, 36, 44, 49], an assumption which is satisﬁed by several interesting families, including quasiconvex-concave families or starconvex-concave families.”
Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile (2018, background): “First, to the best of our knowledge, its first antecedent is a gradient condition examined by Bottou ( ) in the context of nonlinear programming; we borrow the term “coherence” from the more recent paper of Zhou et al. ( ) (who actually used the term to describe strict coherence).”
On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes (2018, background): “A weak condition for almost sure convergence to the global optimum of non-convex functions was proposed in Bottou (1998) and recently independently repro-posed in Zhou et al. (2017).”
On the Convergence of Weighted AdaGrad with Momentum for Training Deep Neural Networks (2018, background): “The convergence and convergence rate of vanilla SGD have been established [3, 13, 45].”
Distributed Asynchronous Optimization with Unbounded Delays: How Slow Can You Go? (2018, methodology): “Our analysis relies on several novel ideas from the theory of stochastic approximation that were leveraged in a series of recent papers to examine descent schemes in a game-theoretic/continuous-time setting (Zhou et al., 2017b; Mertikopoulos & Staudigl, 2018a;b; Mertikopoulos & Zhou, 2018).”
Finite-Time Last-Iterate Convergence for Multi-Agent Learning in Games (2020, methodology): “…literature [Krichene et al., 2015, Lam et al., 2016, Zhou et al., 2017c, Palaiopanos et al., 2017, Zhou et al., 2017b, Mertikopoulos et al., 2017, Zhou et al., 2017a, 2020a, 2018, Zhou et al., Mertikopoulos and Zhou, 2019] has devoted the eﬀorts to obtaining last-iterate convergence results.”
Knowledge Distillation Beyond Model Compression (2020, methodology): “FSP Stage 1: Initialization Stage 2: Standard Blocks 1,2,3 Stage 1: W = [1, 3, 1] Stage 2: α = 0.”
Noise as a Resource for Learning in Knowledge Distillation (2020, background): “Previous studies also showed that noise is crucial for non-convex optimization [26, 32, 58, 63].”
Optimistic Dual Extrapolation for Coherent Non-monotone Variational Inequalities (2021, background): “Assumption 3 is slightly weaker than the variational coherence assumption [41, 42] or the generalized monotone assumption [7].”
Stochastic Gradient/Mirror Descent: Minimax Optimality and Implicit Regularization (2018, background): “Stochastic Mirror Descent (SMD) (Nemirovskii et al., 1983; Beck & Teboulle, 2003; Cesa-Bianchi et al., 2012; Zhou et al., 2017) is one of the most widely used family of algorithms for stochastic optimization, which includes SGD as a special case.”