Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality

By Hédy Attouch, Jérôme Bolte, Patrick Redont, Antoine Soubeyran. 2008

1,445 citations recorded by the source corpus.

We study the convergence properties of an alternating proximal minimization algorithm for nonconvex structured functions of the type: L(x,y)=f(x)+Q(x,y)+g(y), where f and g are proper lower semicontinuous functions, defined on Euclidean spaces, and Q is a smooth function that couples the variables x and y. The algorithm can be viewed as a proximal regularization of the usual Gauss-Seidel method to minimize L. We work in a nonconvex setting, just assuming that the function L satisfies the Kurdyka-Łojasiewicz inequality. An entire section illustrates the relevancy of such an assumption by giving examples ranging from semialgebraic geometry to “metrically regular” problems. Our main result can be stated as follows: If L has the Kurdyka-Łojasiewicz property, then each bounded sequence generated by the algorithm converges to a critical point of L. This result is completed by the study of the convergence rate of the algorithm, which depends on the geometrical properties of the function L around its critical points. When specialized to $Q(x,y)=\Vert x-y \Vert ^2$ and to f, g indicator functions, the algorithm is an alternating projection mehod (a variant of von Neumann's) that converges for a wide class of sets including semialgebraic and tame sets, transverse smooth manifolds or sets with “regular” intersection. To illustrate our results with concrete problems, we provide a convergent proximal reweighted l1 algorithm for compressive sensing and an application to rank reduction problems.

Subfields: Computational Mechanics, Mathematical Physics, Numerical Analysis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Mathematical Physics

In Numerical Analysis

Citations in the corpus

Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, background): “The essence of GKT is that it reformulates FL as an alternating minimization (AM) approach [14, 15, 16, 17, 18, 19], which optimizes two random variables (the edge model and the server model) by alternatively fixing one and optimizing another.”
Coordinate descent algorithms (2015, methodology): “Bouman and Sauer [7] discuss an application to positron emission tomography (PET) in which the objective has the form (2) where f is smooth and convex and Ω is a sum of terms of the form |xj − xl| for some pairs of components (j, l) of x and some q ∈ [1, 2].”
Calculus of the Exponent of Kurdyka–Łojasiewicz Inequality and Its Applications to Linear Convergence of First-Order Methods (2016, methodology): “ent) of the potential function. The KL property has been applied to analyzing local convergence rate of various rst-order methods for a wide variety of problems by many researchers; see, for example, [2, 13, 23, 42]. In these studies, a proto-typical theorem on convergence rate takes the following f”
Fault Tolerance in Iterative-Convergent Machine Learning (2018, background): “In fact, the assumption of strong convexity can be substantially relaxed to include various nonconvex problems (Xu and Yin, 2017; Attouch et al., 2010).”
Modern regularization methods for inverse problems (2018, background): “It is important to emphasize that even for nonsmooth, nonconvex optimization there is a vast amount of recent publications, ranging from forward-backward, respectively proximaltype, schemes [8, 9, 10, 49, 50], over linearized proximal schemes [365, 47, 366, 298], to inertial methods [299, 309], prim”
Fusing Hyperspectral and Multispectral Images via Coupled Sparse Tensor Factorization (2018, background): “[46], [47]) to solve (15), which can be guaranteed to converge to a critical point under certain conditions.”
Proximal Algorithms in Statistics and Machine Learning (2015, methodology): “The proximal operator of ` has a closed-form, multi-valued solution and convergence results are available for proximal methods in Marjanovic and Solo [2013] and Attouch et al. [2013]. For this example, we choose the former approach.”
Proximal alternating linearized minimization for nonconvex and nonsmooth problems (2013, background): “, [2]), thus the truly relevant aspect of this property is when ū is critical, i.”
A Block Coordinate Descent Method for Regularized Multiconvex Optimization with Applications to Nonnegative Tensor Factorization and Completion (2013, methodology): “Our analysis here was motivated by [4], which applies the inequality to establish the global convergence of the alternating proximal point method — the special case of BCD with two blocks and using update (1.3b).”
On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift (2019, neutral): “…domination property if for all θ ∈ Θ, f(θ⋆)− f(θ) = O(G(θ)), where θ⋆ ∈ argmaxθ′∈Θ f(θ′) and where G(θ) is some suitable scalar notion of first-order stationarity, which can be considered a measure of how large the gradient is (see [Karimi et al., 2016, Bolte et al., 2007, Attouch et al., 2010]).”
Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge (2020, background): “The essence of FedGKT is that it reformulates FL as an alternating minimization (AM) approach [13, 14, 15, 16, 17, 18], which optimizes two random variables (the edge model and the server model) by alternatively fixing one and optimizing another.”
Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization (2019, methodology): “The KŁ inequality has been used to show the convergence of several ﬁrst-order optimization meth-ods towards critical points (Attouch and Bolte, 2009; Attouch et al., 2010; 2013; Bolte et al., 2014; Frankel et al., 2015; Li et al., 2017).”
iPiano: Inertial Proximal Algorithm for Nonconvex Optimization (2014, methodology): “A big class of functions that have the KL property is given by real semialgebraic functions [4].”