DiSCO: Distributed Optimization for Self-Concordant Empirical Loss

By Yuchen Zhang, Xiao Lin. 2015

232 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Federated Learning: Strategies for Improving Communication Efficiency (2016, background): “The above framework differs from conventional distributed machine learning [17, 11, 19, 22, 5, 4] due to the the large number of clients, highly unbalanced and non-i.”
Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “As shown in Zhang and Lin (2015), for all t ≤ 1 / 6, ω ∗ ( t ) ≤ 1 .”
Federated Learning: Collaborative Machine Learning without Centralized Training Data (2022, neutral): “Further distributed optimization methods are DANE [25], its accelerated variant AIDE [21], and DiSCO [27].”
Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “An orthogonal one is to pay more local computation for less communication, e.g., one-shot aggregation [53, 54, 16, 21, 46], primal-dual algorithms [37, 38, 7] and distributed Newton methods [33, 52, 29, 35, 25].”
The Non-IID Data Quagmire of Decentralized Machine Learning (2019, methodology): “These works reduce communication overhead by proposing algorithm-speciﬁc approaches, such as solving a dual problem (Shalev-Shwartz & Zhang, 2013; Takác et al., 2013; Jaggi et al., 2014) or employing a different optimization algorithm (Rakhlin et al., 2012; Zhang & Lin, 2015).”
On the Convergence of FedAvg on Non-IID Data (2019, methodology): “Similar to the centralized parallel optimization (Jakovetic, 2013; Li et al., 2014a;b; Shamir et al., 2014; Zhang and Lin, 2015; Meng et al., 2016; Reddi et al., 2016; Richtárik and Takác, 2016; Smith et al., 2016; Zheng et al., 2016; Shusen Wang et al., 2018), FL lets the user devices (aka worker n”
Byzantine Stochastic Gradient Descent (2018, background): “ny algorithm can achieve, no matter how many samples are taken. Distributed SGD. There is an extremely rich literature studying the scalability and communication requirements of distributed SGD, e.g. [1, 9, 23, 24, 31, 32, 35, 40], in settings where all the workers are correct. Since none of these a”
Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates (2018, methodology): “More sophisticated algorithms have been proposed in order to achieve better accuracy than the one-round approach while maintaining lower communication costs; examples include DANE (Shamir et al., 2014), Disco (Zhang and Lin, 2015), distributed SVRG (Lee et al., 2015) and their variants (Reddi et al.”
AIDE: Fast and Communication Efficient Distributed Optimization (2016, methodology): “These methods are D ANE [20], D ISCO [23] and C OCOA + [11, 5, 12, 21].”
Bayesian Nonparametric Federated Learning of Neural Networks (2019, background): “Yet others explore distributed optimization with a speciﬁc emphasis on communication ef-ﬁciency (Zhang et al., 2013; Shamir et al., 2014; Yang, 2013; Ma et al., 2015; Zhang & Lin, 2015).”
Federated Optimization: Distributed Machine Learning for On-Device Intelligence (2016, methodology): “The DiSCO algorithm [105] of Zhang and Xiao is based on inexact damped Newton method.”
Gradient Sparsification for Communication-Efficient Distributed Optimization (2017, background): “The existing works on distributed machine learning mainly focus on two directions: 1) how to design communication efficient algorithms to reduce the round of communications among workers [43, 31, 13, 27, 42], and 2) about how to use large mini-batches without compromising the convergence speed [21, ”
Adding vs. Averaging in Distributed Primal-Dual Optimization (2015, background): “The above class of problems includes many standard problems of wide interest in machine learning, statistics and signal processing, including Support Vector Machines, as well as regularized linear and logistic regression, ordinal regression, and others.”
Distributed optimization with arbitrary local solvers (2015, background): “On typical real-world systems, communicating data between machines is several orders of magnitude slower than reading data from main memory, e.g., when leveraging commodity hardware.”
LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning (2018, methodology): “Roughly speaking, algorithms in (Smith et al., 2017; Jaggi et al., 2014; Ma et al., 2017; Shamir et al., 2014; Zhang and Lin, 2015) reduce communication by increasing local computation (relative to GD), while our method does not increase local computation.”
Memory and Communication Efficient Distributed Stochastic Optimization with Minibatch Prox (2017, background): “In fact, this disadvantage is shared by previously known communication-efficient distributed optimization algorithms, including DANE, DiSCO, and AIDE.”
GIANT: Globally Improved Approximate Newton Method for Distributed Optimization (2017, methodology): “However, in comparison, GIANT has some major advantages over the prior works DANE, AIDE, CoCoA (Smith et al., 2016), and DiSCO (Zhang and Lin, 2015).”
Stochastic, Distributed and Federated Optimization for Machine Learning (2017, methodology): “The DiSCO algorithm [199] of Zhang and Xiao is based on inexact damped Newton method.”
Communication-Efficient Federated Deep Learning With Layerwise Asynchronous Model Update and Temporally Weighted Aggregation (2019, methodology): “Such a framework appears to be similar to distributed machine learning algorithm [7], [8], [9], [10], [11], [12].”