Efficient Distributed Learning with Sparsity

By Jialei Wang, Mladen Kolar, Nati Srebro, Tong Zhang. 2016

178 citations recorded by the source corpus.

We propose a novel, efficient approach for distributed sparse learning in high-dimensions, where observations are randomly partitioned across machines. Computationally, at each round our method only requires the master machine to solve a shifted ell_1 regularized M-estimation problem, and other workers to compute the gradient. In respect of communication, the proposed approach provably matches the estimation error bound of centralized methods within constant rounds of communications (ignoring logarithmic factors). We conduct extensive experiments on both simulated and real world datasets, and demonstrate encouraging performances on high-dimensional regression and classification tasks.

Subfields: Computational Mechanics, Computer Networks and Communications, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Jason D. Lee
2. Hanlin Tang
3. Ce Zhang
4. Michael I. Jordan
5. Ji Liu
6. Dan Alistarh
7. Ruslan Salakhutdinov
8. Percy Liang

In Computer Networks and Communications

1. Hanlin Tang
2. Ji Liu
3. Ce Zhang
4. Dan Alistarh
5. Liu Ziyin
6. Massih-Reza Amini
7. Ruslan Salakhutdinov
8. Percy Liang

In Statistics and Probability

1. Percy Liang
2. Jason D. Lee
3. Michael I. Jordan
4. Ruslan Salakhutdinov
5. Massih-Reza Amini
6. Ce Zhang
7. Dmitry Grishchenko
8. Yun Yang

Citations in the corpus

• Distributed Learning over Unreliable Networks (2018, methodology): “To save the communication cost, some sparse based distributed learning algorithms is proposed (Shen et al., 2018b; Wu et al., 2018; Wen et al., 2017; McMahan et al., 2016; Wang et al., 2016).”
• Asynchronous Distributed Learning with Sparse Communications and Identification (2018, background): “Typically, one uses (cid:96) 0 or (cid:96) 1 norms, the proximity operator of which are respectively hard or soft thresholding [32, 27].”
• Think Locally, Act Globally: Federated Learning with Local and Global Representations (2020, background): “Recent work has improved the communication efficiency of distributed learning by sparsifying the data and model [76], developing efficient gradient-based methods [77, 21], and compressing the updates [75, 83, 49, 52, 46].”
• Communication-Efficient Distributed Statistical Inference (2016, background): “(We note that during the preparation of this manuscript, we became aware of concurrent work of Wang et al. (2017) who also study a communication-efficient surrogate likelihood.”