Distributed Estimation, Information Loss and Exponential Families

By Qiang Liu, A. Ihler. 2014

61 citations recorded by the source corpus.

Distributed learning of probabilistic models from multiple data repositories with minimum communication is increasingly important. We study a simple communication-efficient learning framework that first calculates the local maximum likelihood estimates (MLE) based on the data subsets, and then combines the local MLEs to achieve the best possible approximation to the global MLE given the whole dataset. We study this framework's statistical properties, showing that the efficiency loss compared to the global setting relates to how much the underlying distribution families deviate from full exponential families, drawing connection to the theory of information loss by Fisher, Rao and Efron. We show that the "full-exponential-family-ness" represents the lower bound of the error rate of arbitrary combinations of local MLEs, and is achieved by a KL-divergence-based combination method but not by a more common linear combination method. We also study the empirical properties of both methods, showing that the KL method significantly outperforms linear combination in practical settings with issues such as model misspecification, non-convexity, and heterogeneous data partitions.

Subfields: Artificial Intelligence, Computer Networks and Communications, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Statistics and Probability

Citations in the corpus

Fast Learning from Distributed Datasets without Entity Matching (2016, background): “Most literature on distributed learning is concerned with l imiting communication and designing optimal strategies for merging models [2, 17]; beside that, previous works focus on horizontal split by observations, with few exceptions [18].”
DUAL-LOCO: Distributing Statistical Estimation Using Random Projections (2015, background): “Other recent work considers solving statistical estimation tasks using a single round of communication [6, 7].”
LOCO: Distributing Ridge Regression with Random Projections (2014, background): “is the average of the local estimates achieves a superlinear speedup whilst retaining an optimal rate of convergence (in the statistical sense) up to a number of workers, Kwhich is problem dependent. Liu and Ihler (2014) address the more general problem of distributed maximum likelihood estimation i”
On the Optimality of Averaging in Distributed Statistical Learning (2014, background): “This stands in contrast to other works that demonstrate how, under different assumptions, parallelization combined with averaging may incur a large error [Liu and Ihler, 2014], or even an unbounded one [Shamir et al.”