On the linear convergence of the alternating direction method of multipliers

By Mingyi Hong, Zhi-Quan Tom Luo. 2012

768 citations recorded by the source corpus.

We analyze the convergence rate of the alternating direction method of multipliers (ADMM) for minimizing the sum of two or more nonsmooth convex separable functions subject to linear constraints. Previous analysis of the ADMM typically assumes that the objective function is the sum of only two convex functions defined on two separable blocks of variables even though the algorithm works well in numerical experiments for three or more blocks. Moreover, there has been no rate of convergence analysis for the ADMM without strong convexity in the objective function. In this paper we establish the global R-linear convergence of the ADMM for minimizing the sum of any number of convex separable functions, assuming that a certain error bound condition holds true and the dual stepsize is sufficiently small. Such an error bound condition is satisfied for example when the feasible set is a compact polyhedron and the objective function consists of a smooth strictly convex function composed with a linear mapping, and a nonsmooth ℓ1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ell _1$$\end{document} regularizer. This result implies the linear convergence of the ADMM for contemporary applications such as LASSO without assuming strong convexity of the objective function.

Subfields: Computational Mechanics, Mathematical Physics, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Mathematical Physics

In Optimization & Training

In Signal Processing

Citations in the corpus

Frank-Wolfe Splitting via Augmented Lagrangian Method (2018, methodology): “In order to avoid getting the same problematic quantity R Y , they use Lemma 3.1 from (Hong and Luo, 2012) (which also appears as Lemma 3.1 in the published version (Hong and Luo, 2017)).”
Communication-Efficient Distributed Optimization using an Approximate Newton-type Method (2013, methodology): “A similar dependence occurs with other recently-proposed algorithms for distributed optimization (e.g. Yang, 2013; Mahajan et al., 2013; Dekel et al., 2012; Cotter et al., 2011; Duchi et al., 2012).”
On the Linear Convergence of the ADMM in Decentralized Consensus Optimization (2013, methodology): “The analysis provide guidelines for networking strategies, objective-function splitting strategies, and algorithm parameter settings to achieve faster convergence.”
LIME: Low-Light Image Enhancement via Illumination Map Estimation (2017, neutral): “Output: Final enhanced result”
Distributed variance regularized Multitask Learning (2016, background): “This connection makes it apparent that the regularization term encourages a small magnitude of the vectors wt (large margin of each SVM) while simultaneously controlling their variance.”
Greedy Direction Method of Multiplier for MAP Inference of Large Output Domain (2017, methodology): “Theorem 1 and 2 are based on the analysis framework of [22] for the Augmented Lagrangian Method, as well as the recent results on the linear convergence of Frank-Wolfe variants [23] and Greedy Coordinate Descent for non-strongly convex functions of the form (13) [24, 25].”
Separable approximations and decomposition methods for the augmented Lagrangian (2013, positive): “The convergence of ADMM has long been established for two-block problems, however the convergence for multiple blocks has only been addressed recently under specific assumptions [14] or through modifications [12,44].”
Dual Decomposed Learning with Factorwise Oracle for Structural SVM of Large Output Domain (2016, methodology): “The analysis leverages recent analysis on the global linear convergence of Frank-Wolfe variants [8] for function of the form (12) with a polyhedral domain, and also the analysis in [5] for Augmented Lagrangian based method.”
Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems (2014, background): “Therefore, most research effort in this direction has been focused on either analyzing problems with additional conditions or showing convergence for variants of the ADMM; see, for example, [26, 27, 28, 29, 30, 31, 32, 33, 34].”
Quantized consensus ADMM for multi-agent distributed optimization (2015, background): “(9) (2)The linear convergence results of [22], [23] do not apply her e.”
Parallel Direction Method of Multipliers (2014, methodology): “Since GSADMM uses new iterates which increases computationcompared to PDMM3, PDMM3 can be faster than GSADMM if the numbers of iterations are close.”