Consensus-based distributed optimization: Practical issues and applications in large-scale machine learning

By Konstantinos I. Tsianos, Sean F. Lawlor, Michael G. Rabbat. 2012

249 citations recorded by the source corpus.

Subfields: Computational Mechanics, Computer Networks and Communications, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Networks and Communications

In Electrical and Electronic Engineering

Citations in the corpus

GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning (2019, neutral): “1-(a), this problem is often solved by locally minimizing f n ( θ n ) at each worker and globally averaging their model parameters θ n ’s (and/or gradients) at a parameter server, thereby yielding the global model parameters θ (Tsianos et al., 2012).”
On the Convergence of Decentralized Gradient Descent (2013, background): “The recent research interest in big data processing also motivates the work of decentralized optimization in machine learning [8, 28].”
Network Topology and Communication-Computation Tradeoffs in Decentralized Optimization (2017, neutral): “Corollary 9.”
Distributed optimization over time-varying directed graphs (2013, background): “The papers [20], [21] proved the convergence of a subgradient algorithm in a directed but fixed topology; implementation of the protocol appears to require knowledge of the graph or of the number of agents.”
DSA: Decentralized Double Stochastic Averaging Gradient Algorithm (2015, methodology): “Although we make no formal assumption, in cases of practical importance the total number of training samples =1 q n is very large, but the numbers of elements q n available at a speciﬁc node are moderate.”
Stochastic Gradient-Push for Strongly Convex Functions on Time-Varying Directed Graphs (2014, background): “a recent strand of the distributed optimization literature which studies effective protocols when interactions between nodes are unidirectional [6], [8], [10], [11], [36], [41].”
Compressed Distributed Gradient Descent: Communication-Efficient Consensus over Networks (2018, background): “, distributed sensing in wireless sensor networks [1]–[4], decentralized machine learning [5], [6], multi-agent robotic systems [7]–[9], smart grids [10], [11], to name just a few.”
Quantized Decentralized Consensus Optimization (2018, background): “Distributed optimization of a sum of convex functions has a variety of applications in different areas including de-centralized control systems [1], wireless systems [2], sensor networks [3], networked multiagent systems [4], multirobot networks [5], and large scale machine learning [6].”
Asynchronous Gossip-Based Random Projection Algorithms Over Networks (2013, methodology): “The gradient-proj ecti n algorithms proposed in the papers [3], [15], [17], [24], [26], [30] assume that the agen ts share a common constraint set and the projection is performed on the whole constraint set a t ach iteration.”
Proximity Without Consensus in Online Multiagent Optimization (2016, background): “Distributed optimization has a rich history in wireless communications [6], sensor networks [7], and machine learning [8].”
DEXTRA: A Fast Algorithm for Optimization Over Directed Graphs (2017, methodology): “These gradient-based methods [20]–[23], [30], [31], [33], restricted by the diminishing step-size, converge relatively slow at O ( ln k √ k ) .”
Balancing Communication and Computation in Distributed Optimization (2017, background): “This is motivated by its wide applicability to many important engineering and scientific domains, such as wireless sensor networks [1]–[4], multivehicle and multirobot networks [5]–[7], smart grids [8], [9], and machine learning [10], [11].”
Nested Distributed Gradient Methods with Adaptive Quantized Communication (2019, background): “Such problems arise in a plethora of applications such as wireless sensor networks [1], [2], multi-vehicle and multirobot networks [3], [4], smart grids [5], [6] and machine learning [7], [8], to mention a few.”
Linear Time Average Consensus on Fixed Graphs and Implications for Decentralized Optimization and Multi-Agent Control (2014, unknown): “…analysis of this problem was given in [58] and the problem has received considerable attention in recent years due its repeated appearance throughout distributed computing and multi-agent control; we mention [58, 59, 52, 70, 72, 42, 41, 79, 81, 80, 84, 34, 35, 8, 51, 37, 76, 17], among many others.”
Efficient Distributed Online Prediction and Stochastic Optimization With Approximate Distributed Averaging (2014, methodology): “A more practical implementation should only require P to be column stochastic and would use push-sum averaging iterations; see [37] for more details.”
Decentralized double stochastic averaging gradient (2015, background): “The formulation in (1) models large scale machine learning problems where elements of the training set are distributed to multiple computational elements and x̃∗ represents an optimal classifier [1], [2].”
Distributed Optimization Over Time-Varying Directed Graphs (2015, unknown): “The papers [30], [31], [32], [29] proposed a distributed subgradient algorithm which is very similar to the one we study here, involving the introduction of subgradients into an information aggregation procedure known as “push-sum.”