Fast optimization via inertial dynamics with closed-loop damping

By Hédy Attouch, Radu Ioan Bot, Ernö Robert Csetnek. Journal of the European Mathematical Society (Print), 2020

58 citations recorded by the source corpus.

In a Hilbert space H, in order to develop fast optimization methods, we analyze the asymptotic behavior, as time t tends to infinity, of inertial continuous dynamics where the damping acts as a closed-loop control. The function f : H → R to be minimized (not necessarily convex) enters the dynamic through it gradient, which is assumed to be Lipschitz continuous on the bounded subsets of H. This gives autonomous dynamical systems with nonlinear damping and nonlinear driving force. We first consider the case where the damping term ∂φ(ẋ(t)) acts as a closed-loop control of the velocity. The damping potential φ : H → R + is a convex continuous function which achieves its minimum at the origin. We show the existence and uniqueness of a global solution to the associated Cauchy problem. Then, we analyze the asymptotic convergence properties of the trajectories generated by this system. To do this, we use techniques from optimization, control theory, and PDE's: Lyapunov analysis based on the decreasing property of an energy-like function, quasi-gradient and Kurdyka-Lojasiewicz theory, monotone operator theory for wave-like equations. Convergence rates are obtained based on the geometric properties of the data f and φ. We put forward minimal hypotheses on the damping term guaranteeing the convergence of trajectories, thus showing the dividing line between strong and weak damping. When f is strongly convex, we give general conditions on the damping potential φ which provide exponential convergence rates. Then, we extend the results to the case where an additional Hessian-driven damping enters the dynamic, which reduces the oscillations. Finally, we consider an inertial system with a closed-loop damping involving jointly the velocityẋ(t) and the gradient ∇f (x(t)). This study naturally leads to similar results for the proximal-gradient algorithms which can be derived by temporal discretization, some of them are studied in the article. In addition to its original results, this work surveys the numerous works devoted in recent years to the interaction between continuous damped inertial dynamics and numerical algorithms for optimization, with the emphasis on autonomous systems, closed-loop adaptive procedures, and convergence rates.

Subfields: Computational Theory and Mathematics, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/d134ee4b6b37aa0dad50307a48ac3de4ce8959f7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Convergence of iterates for first-order optimization algorithms with inertia and Hessian driven damping (2021, background): “Due to the remarkable properties induced by the presence of the Hessiandriven damping, these inertial dynamics and algorithms have been the subject of active recent developments; see the work of [4] on a high resolution perspective, that of [5] for application to deep learning, the papers of [6,7] f”