Strong Data Processing Inequalities and $\Phi $ -Sobolev Inequalities for Discrete Channels

By Maxim Raginsky. 2014

159 citations recorded by the source corpus.

The noisiness of a channel can be measured by comparing suitable functionals of the input and output distributions. For instance, the worst case ratio of output relative entropy to input relative entropy for all possible pairs of input distributions is bounded from above by unity, by the data processing theorem. However, for a fixed reference input distribution, this quantity may be strictly smaller than one, giving the so-called strong data processing inequalities (SDPIs). The same considerations apply to an arbitrary Φ-divergence. This paper presents a systematic study of optimal constants in the SDPIs for discrete channels, including their variational characterizations, upper and lower bounds, structural results for channels on product probability spaces, and the relationship between the SDPIs and the so-called Φ-Sobolev inequalities (another class of inequalities that can be used to quantify the noisiness of a channel by controlling entropy-like functionals of the input distribution by suitable measures of input-output correlation). Several applications to information theory, discrete probability, and statistical physics are discussed.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Statistics and Probability

Citations in the corpus

$f$ -Divergence Inequalities (2015, background): “SUMMARY Since many distance measures of interest fall under the common paradigm of an f -divergence, it is not surprising that bounds on the ratios of various f -divergences are useful in many instances such as proving convergence of probability measures according to various metrics, analysis of rat”
Locally Private Gaussian Estimation (2018, methodology): “We obtain this result by introducing tools from the strong data processing inequality literature [6, 21].”
Information-theoretic analysis of generalization capability of learning algorithms (2017, background): “Moreover, strong data processing inequalities [18] may be used to sharpen these upper bounds on I(S;W ).”
On the Robustness of Information-Theoretic Privacy Measures and Mechanisms (2018, background): “More recent developments about the properties of f -divergence can be found in Raginsky [68], Calmon et al.”