Information-Theoretic Lower Bounds on Bayes Risk in Decentralized Estimation

By Aolin Xu, Maxim Raginsky. 2016

61 citations recorded by the source corpus.

We derive lower bounds on the Bayes risk in decentralized estimation, where the estimator does not have direct access to the random samples generated conditionally on the random parameter of interest, but only to the data received from local processors that observe the samples. The received data are subject to communication constraints due to the quantization and the noisy communication channels from the processors to the estimator. We first derive general lower bounds on the Bayes risk using information-theoretic quantities, such as mutual information, information density, small ball probability, and differential entropy. We then apply these lower bounds to the decentralized case, using strong data processing inequalities to quantify the contraction of information due to communication constraints. We treat the cases of a single processor and of multiple processors, where the samples observed by different processors may be conditionally dependent given the parameter, for noninteractive and interactive communication protocols. Our results recover and improve recent lower bounds on the Bayes risk and the minimax risk for certain decentralized estimation problems, where previously only conditionally independent sample sets and noiseless channels have been considered. Moreover, our results provide a general way to quantify the degradation of estimation performance caused by distributing resources to multiple processors, which is only discussed for specific examples in existing works.

Subfields: Artificial Intelligence, Computer Networks and Communications, Electrical and Electronic Engineering, Statistics and Probability, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Electrical and Electronic Engineering

In Statistics and Probability

In Information Theory

Citations in the corpus

Geometric Lower Bounds for Distributed Parameter Estimation Under Communication Constraints (2018, methodology): “,X) <1 can be desirable. The SDPI may take different forms (e.g., for f-divergences), and it is applied in most works on distributed estimation, e.g., Zhang et al. (2013); Braverman et al. (2016); Xu and Raginsky (2017). The SDPI-based approach turns out to be tight in certain models (e.g., the G”
Inference Under Information Constraints II: Communication Constraints and Shared Randomness (2019, background): “The problem of distributed density estimation, too, has gathered recent interest in various statistical settings [5], [9], [14], [22], [33], [42], [43], [48]–[50].”
Distributed Statistical Estimation of High-Dimensional and Nonparametric Distributions (2018, background): “While trivially I(U ;Y ) ≤ I(U ;X) and I(U ;Y ) ≤ I(X;Y ), neither of these two inequalities are typically sufficient to obtain a good lower bound, and most works [8], [10], [11] rely on strong data processing inequalities of the form”
Lower Bounds for Learning Distributions under Communication Constraints via Fisher Information (2019, background): “Similar to the previous discussions, strong or distributed data-processing inequalities (Duchi et al., 2013; Xu and Raginsky, 2017) are typically used in scenarios where the linear/quadratic dependence on the pricacy parameter ε is tight, and explicit modeling becomes necessary in scenarios where…”
Unified lower bounds for interactive high-dimensional estimation under information constraints (2020, methodology): “Such strong data processing inequalities were used to derive lower bounds for statistical estimation under communication constraints in [35, 11, 32].”
Distributed Signal Detection under Communication Constraints (2020, background): “…estimation has been considered in a number of recent works (Zhang et al., 2013; Garg et al., 2014; Shamir, 2014; Braverman et al., 2016; Xu and Raginsky, 2017; Han et al., 2018a,b; Barnes et al., 2019; Cai and Wei, 2020; Acharya et al., 2019b,d), while distributed testing of distributions in…”