A Geometric Characterization of Fisher Information from Quantized Samples with Applications to Distributed Statistical Estimation

By Leighton Pate Barnes, Yanjun Han, Ayfer Özgür. 2018

12 citations recorded by the source corpus.

Consider the Fisher information for estimating a vector $\theta \in \mathbb {R}^{d}$ from the quantized version of a statistical sample $X \sim f(x|\theta)$. Let M be a k-bit quantization of $X.$ We provide a geometric characterization of the trace of the Fisher information matrix $I_{M}(\theta)$ in terms of the score function $S_{\theta }(X)$. When $k=1$, we exactly solve the extremal problem of maximizing this geometric quantity for the Gaussian location model, which allows us to conclude that in this model, a half-space quantization is the one-bit quantization that maximizes $Tr(I_{M}(\theta))$. Under assumptions on the tail of the distribution of $S_{\theta }(X)$ projected onto any unit vector in $\mathbb {R}^{d}$, we give upper bounds demonstrating how $Tr(I_{M}(\theta))$ can scale with k. We apply these results to find lower bounds on the minimax risk of estimating $\theta $ from multiple quantized samples of X, for example in a distributed setting where the samples are distributed across multiple nodes and each node has a total budget of k-bits to communicate its sample to a centralized estimator. Our bounds apply in a unified way to many common statistical models including the Gaussian location model and discrete distribution estimation, and they recover and generalize existing results in the literature with simpler and more transparent proofs.

Subfields: Computer Networks and Communications, Electrical and Electronic Engineering, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Electrical and Electronic Engineering

In Statistics and Probability

Citations in the corpus

rTop-k: A Statistical Estimation Approach to Distributed SGD (2020, background): “It builds on a geometric characterization of Fisher information from quantized samples, a framework introduced in [35].”
Learning Distributions from their Samples under Communication Constraints (2019, background): “Equivalently, we ask the question: how can we best represent X with k bits so as to maximize the Fisher information it provides about θ? This framework was first introduced by the authors in [1],[2].”
Fisher Information Under Local Differential Privacy (2020, background): “Our Fisher information approach to lower bounds under privacy constraints is motivated by recent results for statistical estimation under communication constraints such as [15], [16], [17].”
Lower Bounds for Learning Distributions under Communication Constraints via Fisher Information (2019, methodology): “This framework was first introduced in Barnes et al. (2018, 2019), and there has been some previous work in analyzing Fisher information from a quantized scalar random variable such as Venkitasubramaniam et al. (2006, 2005); Ribeiro and Giannakis (2005); Lam and Reibman (1993).”
Fisher Information for Distributed Estimation under a Blackboard Communication Protocol (2019, methodology): “With the exception of our prior work [1], all prior work on related problems (including [5], [6], [7], [3], [2], [13]) build on an alternative well-established technique for developing minimax lower bounds on statistical estimation problems that relies on converting the estimation problem to a caref”