Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers

By Stephen P. Boyd, Neal Parikh, Eric King-Wah Chu, Eric Chu, Borja Peleato, Borja Peleato, Jonathan Eckstein, Jonathan Eckstein. 2011

8,037 citations recorded by the source corpus.

Subfields: Optimization & Training, Mathematical Optimization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Mathematical Optimization

Citations in the corpus

Federated Optimization in Heterogeneous Networks (2018, neutral): “…particularly in data center settings, has motivated the development of numerous distributed optimization methods in the past decade (see, e.g., Boyd et al., 2010; Dekel et al., 2012; Dean et al., 2012; Zhang et al., 2013; Li et al., 2014a; Shamir et al., 2014; Reddi et al., 2016; Zhang et…”
Harnessing Wireless Channels for Scalable and Privacy-Preserving Federated Learning (2020, positive): “Compared to the existing analog FL algorithms [9–11] based on first-order methods such as GD and SGD, A-FADMM is a second-order method providing faster convergence [15, 16].”
Robust Aggregation for Federated Learning (2019, neutral): “Recent work includes a primal-dual framework called COCOA [44, 60] and its decentralized version [25], as well as asynchronous incremental algorithms [38] and distributed optimization in networks [14, 52, 57].”
GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning (2019, positive): “In contrast to the Gauss-Seidel ADMM in (Boyd et al., 2011), GADMM allows all the head (tail) workers to update their parameters in parallel and still converges to the optimal solution for convex functions as will be shown later in this paper.”
Robust and Communication-Efficient Collaborative Learning (2019, neutral): “The most popular ﬁrst-order choices for the convex setting are distributed gradient descent-type methods (Nedic and Ozdaglar, 2009; Jakovetic et al., 2014; Yuan et al., 2016; Qu and Li, 2017), augmented Lagrangian algorithms (Shi et al., 2015a,b; Mokhtari and Ribeiro, 2016), distributed variants of ”
DJAM: Distributed Jacobi Asynchronous Method for Learning Personal Models (2018, positive): “This parameter, which we refer to as ρ , is known to affect noticeably the convergence speed of ADMM.”
Federated Learning: Challenges, Methods, and Future Directions (2019, neutral): “These methods drastically improve performance in practice, and have been shown to achieve orders-of-magnitude speedups over traditional mini-batch methods or distributed approaches like ADMM [14] in real-world data center environments.”
CoCoA: A General Framework for Communication-Efficient Distributed Optimization (2016, positive): “Alternating Direction Method of Multipliers (ADMM) (Boyd et al., 2010) is a popular method that lends itself naturally to the distributed environment.”
Communication-Efficient Distributed Optimization using an Approximate Newton-type Method (2013, neutral): “In contrast, while more data improves the ADMM accuracy after a ﬁxed number of iterations, the convergence rate is slower and does not improve with the data size 5 .”
Federated Optimization: Distributed Machine Learning for On-Device Intelligence (2016, neutral): “Another class of algorithms relevant for this work is Alternating Direction Method of Multipliers (ADMM) [14, 28].”
Data Synthesis based on Generative Adversarial Networks (2018, neutral): “For the difference between range-based and point-based search, refer to [13].”
Additive models with trend filtering (2017, positive): “2 of Boyd et al. (2011), the authors show that if (i) the criterion decomposes as a sum of closed and convex functions, and (ii) strong duality holds, then the ADMM algorithm produces dual iterates that converge to optimal dual solutions.”
Principal Components and Regularized Estimation of Factor Models (2017, neutral): “10 Algorithms that solve (10) include Augmented Lagrange Multiplier and Accelerated Proximal Gradient (Lin et al. (2013)), ADMM (Boyd et al. (2010)), and CVX (Grant and Boyd (2015)). has solution and that L = A B ′ also solves (10).”
Fastfood: Approximate Kernel Expansions in Loglinear Time (2014, neutral): “of the above algorithms. Having an explicit function expansion is extremely benecial from an optimization point of view. Recent advances in both online (Ratli et al.,2007) and batch (Teo et al.,2010;Boyd et al., 2010) subgradient algorithms summarily rely on the ability to compute gradients in the f”
Distributed optimization with arbitrary local solvers (2015, positive): “ADMM also comes with an additional penalty parameter balancing between the equality constraint on the primal variable vector w and the original optimization objective [9], which is typically hard to tune in many applications.”
An asynchronous parallel stochastic coordinate descent algorithm (2013, neutral): “The alternating direction method-of-multiplier (ADMM) framework [Boyd et al., 2011] can also be implemented in paralle.”