Distributed Dual Averaging In Networks

By John C. Duchi, Alekh Agarwal, Martin J. Wainwright. 2010

76 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Little Is Enough: Circumventing Defenses For Distributed Learning (2019, methodology): “…has become a wide-spread framework for large-scale model training (Dean et al., 2012; Li et al., 2014a;b; Baydin et al., 2017; Zhang et al., 2017; Agarwal et al., 2010; Recht et al., 2011), in which a server is leveraging the compute power of many devices by aggregating local models trained on…”
DRACO: Byzantine-resilient Distributed Training via Redundant Gradients (2018, background): “Distributed and parallel implementations of stochastic optimization algorithms have become the de facto standard in large-scale model training (Li et al., 2014; Recht et al., 2011; Zhang et al., 2015; Agarwal et al., 2010; Abadi et al., 2016; Chen et al., 2015; Paszke et al., 2017a; Chilimbi et al.,”
Optimal Distributed Online Prediction Using Mini-Batches (2010, methodology): “In another related work, Duchi et al. (2010) present a distributed dual averaging method for optimization over networks.”
Distributed dual averaging method for multi-agent optimization with quantized communication (2012, methodology): “There has been much recent research interest in developing distributed methods for solving convex optimization problems in networked-systems [1–7].”
Optimal Distributed Online Prediction (2011, background): “(2010); Duchi et al. (2010). Despite the large activity in this area, the existing guarantees for all the algorithms we are aware of are either purely asymptotic (without rate of convergence in the optimization setting), or are not significantly better than the trivial solution that has an O( √ km) ”
The Effect of Network Width on the Performance of Large-batch Training (2018, background): “(f) Cifar10, ResNet18, FC 102 103 Batch Size 25 50 75 Num Blocks:[3, 3, 3, 3] Num Blocks:[5, 5, 5, 5] Num Blocks:[6, 6, 6, 6]”
Parameter Server for Distributed Machine Learning (2013, background): “Of course, beyond these general frameworks, numerous systems have been developed that target specific applications, such as [3, 13, 24, 22, 28, 10, 15].”
Online Learning: A Comprehensive Survey (2018, background): “In decentralized networks, however, there is no master and each node can only communicate with its neighbors (Agarwal et al., 2010; Mota et al., 2013).”
Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent (2011, methodology): “This recursion underlies many convergence proofs for SGD where ak denotes the distance to the optimal solution after k iterations.”
Draco: Robust Distributed Training against Adversaries (2018, background): “In recent years, the computational paradigm for large-scale machine learning (ML) has started to shift towards massively large distributed systems, sometimes comprising individually small, low-cost compute nodes [1, 7, 9, 10, 12].”
Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers (2011, unknown): “See [54] for a recent discussion of distributed methods for graph-structured optimization problems.”
Towards a unified architecture for in-RDBMS analytics (2012, methodology): “oblem of dierent sampling strategies for stochastic gradient algorithms. There has been a good deal of work in the machine learning community to create several clever parallelization schemes for IGD [12,18,20,28,52]. Our work builds on this work to study those methods that are ideally suited for an ”