Heavy-ball method in nonconvex optimization problems

By S. Zavriev, F. V. Kostyuk. 1993

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization (2019, background): “However, even if this conjecture is true, the hyper-parameter tuning can be extremely time-consuming. optimization is proven in (Zavriev & Kostyuk, 1993) and the O(1/ √ T ) convergence rate for stochastic non-convex optimization is recently shown in (Yan et al., 2018).”
Randomized Iterative Methods for Linear Systems: Momentum, Inexactness and Gossip (2019, methodology): “Extensions of the heavy ball method have been recently proposed in the proximal setting [145], non-convex setting [146, 210] and for distributed optimization [63].”
iPiano: Inertial Proximal Algorithm for Nonconvex Optimization (2014, methodology): “In [45], it has been generalized to the case of smooth nonconvex functions.”
An introduction to continuous optimization for imaging (2016, methodology): “Likewise, inertial methods can be used and are generally convergent (Zavriev and Kostyuk 1991) if ∇f is L-Lipschitz and with suitable assumptions which ensure the boundedness of the trajectories.”
Global convergence of the Heavy-ball method for convex optimization (2014, background): “[18] provided sufficient conditions for the Heavy-ball trajectories to converge to a stationary point.”
Techniques for Gradient-Based Bilevel Optimization with Non-smooth Lower Level Problems (2016, methodology): “As pointed out in the introduction, we focus on gradient based methods, such as gradient descent, L-BFGS [24], non-linear conjugate gradient [18,1], Heavy-ball method [39], iPiano [28], and others, for solving the bilevel optimization problem.”
Convergence Guarantees for RMSProp and ADAM in Non-Convex Optimization and an Empirical Comparison to Nesterov Acceleration (2018, methodology): “Momentum methods like HB and NAG have been shown to have superior convergence properties compared to gradient descent in the deterministic setting both for convex and non-convex functions [29, 33, 48, 30, 31, 16].”
Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform (2020, unknown): “We generalize the heavy ball (HB) momentum update (Ghadimi et al., 2015; Zavriev & Kostyuk, 1993) in the kth optimization step1 to the Stiefel manifold.”