Communication-Efficient Distributed Statistical Inference

By Michael I. Jordan, Jason D. Lee, Yun Yang. 2016

520 citations recorded by the source corpus.

ABSTRACT We present a communication-efficient surrogate likelihood (CSL) framework for solving distributed statistical inference problems. CSL provides a communication-efficient surrogate to the global likelihood that can be used for low-dimensional estimation, high-dimensional regularized estimation, and Bayesian inference. For low-dimensional estimation, CSL provably improves upon naive averaging schemes and facilitates the construction of confidence intervals. For high-dimensional regularized estimation, CSL leads to a minimax-optimal estimator with controlled communication cost. For Bayesian inference, CSL can be used to form a communication-efficient quasi-posterior distribution that converges to the true posterior. This quasi-posterior procedure significantly improves the computational efficiency of Markov chain Monte Carlo (MCMC) algorithms even in a nondistributed setting. We present both theoretical analysis and experiments to explore the properties of the CSL approximation. Supplementary materials for this article are available online.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning (2019, background): “For a large scale network, the communication cost often becomes dominant compared to the computation time, calling for communication eﬃcient distributed optimization (Zhang et al., 2012; McMahan et al., 2017; Park et al., 2018; Jordan et al., 2018; Liu et al., 2019; Sriranga et al., 2019).”
Distributed Statistical Machine Learning in Adversarial Settings (2017, background): “Many efficient distributed machine learning algorithms [6, 13] and system implementations [24, 28, 10, 19] have been proposed and studied.”
LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning (2018, neutral): “Communication-efficient learning algorithms have gained popularity [20, 21].”
Broadband Analog Aggregation for Low-Latency Federated Edge Learning (2018, background): “While computing speeds are growing rapidly, wireless transmission of highdimensional data by many devices suffers from the scarcity of radio resources and hostility of wireless channels, resulting in a communication bottleneck for fast edge learning [5], [6].”
Federated Learning in Mobile Edge Networks: A Comprehensive Survey (2019, background): “• Joint Radio and Computation Resource Management: Even though computation capabilities of mobile devices have grown rapidly, many devices still face a scarcity of radio resources [115].”
One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning: Design and Convergence Analysis (2020, methodology): “The loss function measuring the model error is defined following the existing literature as follows [3], [5], [30], [31].”
Robust Distributed Gradient Descent with Arbitrary Number of Byzantine Attackers (2018, background): “The design of distributed optimization algorithms has attracted significant recent research interests [1, 2, 3, 4, 5].”
One-Shot Federated Learning: Theoretical Limits and Algorithms to Achieve Them (2019, methodology): “This is in contrast to the works of (Zhang et al., 2012) and (Jordan et al., 2018) that assume Lipschitz continuity of second or third derivatives.”
Order Optimal One-Shot Distributed Learning (2019, methodology): “, 2012] and [Jordan et al., 2018] that assume Lipschitz continuity of second or third derivatives.”
Communication Efficient Parallel Algorithms for Optimization on Manifolds (2018, methodology): “In this work, we focus on generalizing a particular parallel inference framework, the Iterative Local Estimation Algorithm (ILEA) [12], to manifolds.”