Communication-efficient sparse regression: a one-shot approach

By Jason D. Lee, Yuekai Sun, Qiang Liu, Jonathan E. Taylor. 2015

70 citations recorded by the source corpus.

We devise a one-shot approach to distributed sparse regression in the high-dimensional setting. The key idea is to average "debiased" or "desparsified" lasso estimators. We show the approach converges at the same rate as the lasso as long as the dataset is not split across too many machines. We also extend the approach to generalized linear models.

Subfields: Artificial Intelligence, Computational Mechanics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jason D. Lee
2. Michael I. Jordan
3. Nati Srebro
4. Peter Richtárik
5. Francis R. Bach
6. Tong Zhang
7. Jakub Konecný
8. Jialei Wang

In Computational Mechanics

1. Jason D. Lee
2. Peter Richtárik
3. Tong Zhang
4. Michael I. Jordan
5. Nati Srebro
6. Jialei Wang
7. Francis R. Bach
8. Jakub Konecný

In Statistics and Probability

1. Jason D. Lee
2. Nati Srebro
3. Michael I. Jordan
4. Tong Zhang
5. Michael Minyi Zhang
6. Lizhen Lin
7. Bayan Saparbayeva
8. Jialei Wang

Citations in the corpus

• Distributed Multi-Task Learning (2015, background): “For example, each machine serves a different geographical location, or each is at a different hospital or school with different characteristics.”
• MSplit LBI: Realizing Feature Selection and Dense Estimation Simultaneously in Few-shot and Zero-shot Learning (2018, background): “Elham-ifar and Vidal (2009), few-shot learning Lee et al. (2015) and zero-shot learning Kodirov et al. (2015).”
• Communication Efficient Parallel Algorithms for Optimization on Manifolds (2018, background): “The general scheme in the frequentist setting is to divide the data into subsets, obtain estimates from each subset which are combined to form an ultimate estimate for inference [9, 30, 17].”
• Distributed Statistical Estimation and Rates of Convergence in Normal Approximation (2017, background): “…acknowledge the works by Mcdonald et al. (2009); Zhang, Wainwright and Duchi (2012); Fan, Han and Liu (2014); Shafieezadeh-Abadeh, Esfahani and Kuhn (2015); Battey et al. (2015); Duchi et al. (2014); Lee et al. (2015); Cheng and Shang (2015); Rosenblatt and Nadler (2016); Zinkevich et al. (2010).”
• Communication-Efficient Distributed Statistical Inference (2016, methodology): “Lee et al. (2015a) extended this idea to high-dimensional sparse linear regression by combining local debi-ased Lasso estimates (van de Geer et al. 2014).”
• Efficient Distributed Learning with Sparsity (2016, methodology): “Unfortunately for high-dimensional problems, the Hessian is no longer invertible, and a `1 regularization is added to make the solution well behaved.”