Asynchronous Distributed ADMM for Consensus Optimization

By Ruiliang Zhang, James T. Kwok. 2014

407 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Mathematical Optimization

Citations in the corpus

Asynchronous Decentralized Parallel Stochastic Gradient Descent (2017, background): “9/n (where n is the number of workers) as recommended in Zhang et al. [2015] for EAMSGD.”
Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent (2017, background): “…and Nedic, 2011, Sundhar Ram et al., 2010] or implicitly indicate that decentralized algorithms were much worse than centralized algorithms in terms of computational complexity and total communication complexity [Aybat et al., 2015, Lan et al., 2017, Ram et al., 2010, Zhang and Kwok, 2014].”
Communication-Efficient Learning of Deep Networks from Decentralized Data (2016, background): “, [41]) relax the IID assumption, but are still not a good fit for communication-constrained optimization over very many clients.”
Deep learning with Elastic Averaging SGD (2014, methodology): “The standard methods for solving the consensus problem such as ADMM Boyd et al. (2011); Zhang & Kwok (2014); Wei & Ozdaglar (2012) therefore no longer apply.”
Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization (2015, methodology): “…applied to speedup many state-of-the-art optimization algorithms including stochastic gradient [Niu et al., 2011, Agarwal and Duchi, 2011, Zhang et al., 2014, Feyzmahdavian et al., 2015, Paine et al., 2013, Mania et al., 2015], stochastic coordinate descent [Avron et al., 2014, Liu et…”
Improving the Privacy and Accuracy of ADMM-Based Distributed Algorithms (2018, methodology): “…consist of subgradient-based algorithms (Nedic et al., 2008; Nedic & Ozdaglar, 2009; Lobel & Ozdaglar, 2011), ADMM-based algorithms (Wei & Ozdaglar, 2012; Ling & Ribeiro, 2014; Shi et al., 2014; Zhang & Kwok, 2014; Ling et al., 2016), and composite of subgradient and ADMM (Bianchi et al., 2014).”
A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Order (2016, methodology): “We encourage readers to refer to recent literatures, for example, Hong [14], Zhang and Kwok [35].”
Ensemble-Compression: A New Method for Parallel Training of Deep Neural Networks (2016, methodology): “duced by MA does not have guarantee on its performance improvement over the local models. Then, we propose using ensemble to perform the model aggregation. Actually, MA (and its variants such as ADMM [22, 20]) was originally proposed for convex optimization. If the model is convex w.r.t. the paramet”
Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems (2014, methodology): “In practice, the algorithm often exhibits faster convergence than traditional primal-dual type algorithms such as the dual ascent algorithm [16, 17, 18] or the method of multipliers [19].”
Unwrapping ADMM: Efficient Distributed Computing via Transpose Reduction (2015, background): “…of ADMM have subsequently been proposed, including specialized variants for decentralized systems (Mota et al., 2013), asynchronous updates (Zhang & Kwok, 2014; Ouyang et al., 2013), inexact solutions to subproblems (Chang et al., 2015), and online/stochastic updates (Ouyang et al., 2013).”
Distributed Learning with Sparse Communications by Identification (2018, methodology): “ynchronous Distributed Optimization. There is a vast literature on distributed optimization methods without shared memory, based a variety of algorithms such as the ADMM, proximal gradient, SAGA etc. [35, 19, 1, 25, 5]. However, these methods usually rely on restrictive assumptions on the computing ”
Multi-Agent Distributed Lifelong Learning for Collective Knowledge Acquisition (2017, background): “Distributed Machine Learning: There has been a growing interest in developing scalable learning algorithms using distributed optimization [40], motivated by the emergence of big data [6], security and privacy constraints [37], and the notion of cooperative and collaborative learning agents [8].”
DP-ADMM: ADMM-Based Distributed Learning With Differential Privacy (2018, methodology): “As pointed out in recent studies [1], [2], existing approaches to decentralizing an optimization problem mainly consist of subgradient-based algorithms [3], [4], alternating direction method of multipliers (ADMM) based algorithms [5]–[8], and composite of sub-gradient descent and ADMM [9].”
Asynchronous Distributed Learning with Sparse Communications and Identification (2018, background): “There is a vast literature on distributed optimization methods without shared memory, based a variety of algorithms such as the ADMM, proximal gradient, SAGA etc. [35, 19, 1, 25, 5].”
A Distributed, Asynchronous and Incremental Algorithm for Nonconvex Optimization: An ADMM Based Approach (2014, background): “The second major difference with the existing literature is about the tasks performed by the distributed nodes: in [37], [44], [45] each node directly optimizes the augmented Lagrangian, while here each node computes the gradient of their respective component functions.”
Distributed stochastic optimization for deep learning (thesis) (2016, positive): “To meet this challenge, many distributed stochastic optimization methods have been proposed in literature, e.g. the mini-batch SGD, the asynchronous SGD, and the ADMM -based methods.”
Fast Asynchronous Parallel Stochastic Gradient Descent: A Lock-Free Approach with Convergence Guarantee (2016, background): “…(SGD) and its variants (Xiao 2009; Duchi and Singer 2009; Roux, Schmidt, and Bach 2012; Johnson and Zhang 2013; Mairal 2013; Shalev-Shwartz and Zhang 2013; Liu et al. 2014; Nitanda 2014; Zhang and Kwok 2014) have recently attracted much attention to solve machine learning problems like that in (1).”