On Pinsker's and Vajda's Type Inequalities for Csiszár's $f$ -Divergences

By Gustavo L. Gilardoni. 2006

115 citations recorded by the source corpus.

Let <i>D</i> and <i>V</i> denote respectively Information Divergence and Total Variation Distance. Pinsker's and Vajda's inequalities are respectively <i>D</i> ≥ [ 1/ 2] <i>V</i>² and <i>D</i> ≥ log[( 2+<i>V</i>)/( 2-<i>V</i>)] - [( 2<i>V</i>)/( 2+<i>V</i>)]. In this paper, several generalizations and improvements of these inequalities are established for wide classes of <;i>f<;/i>-divergences. First, conditions on <i>f</i> are determined under which an <i>f</i>-divergence <i>Df</i> will satisfy <i>Df</i> ≥ <i>cf V</i>² or <i>Df</i> ≥ <i>c</i>_2,<i>f V</i>² + <i>c</i>_4,<i>f V</i>⁴, where the constants <i>cf</i>, <i>c</i>_2,<i>f</i> and <i>c</i>_4,<i>f</i> are best possible. As a consequence, lower bounds in terms of <i>V</i> are obtained for many well known distance and divergence measures, including the χ² and Hellinger's discrimination and the families of Tsallis' and Rényi's divergences. For instance, if <i>D</i>_(α) (<i>P</i>||<i>Q</i>) = [α(α-1)]^-1 [∫<i>p</i>^α<i>q</i>^1-α<i>d</i>μ-1] and ℑ_α (<i>P</i>||<i>Q</i>) = (α-1)^-1 log[∫<i>p</i>^α<i>q</i>^1-α<i>d</i>μ] are respectively the relative information of type α and the Rényi's information gain of order α, it is shown that <i>D</i>_(α) ≥ [ 1/ 2] <i>V</i>² + [ 1/ 72] (α+1)(2-α) <i>V</i>⁴ whenever -1 ≤ α ≤ 2, α ≠ 0,1 and that ℑ_α ≥ [( α)/ 2] <i>V</i>² + [ 1/ 36] α(1 + 5 α- 5 α² ) <i>V</i>⁴ for 0 <; α <; 1. In a somewhat different direction, and motivated by the fact that these Pinsker's type lower bounds are accurate only for small variation (<i>V</i> close to zero), lower bounds for <i>Df</i> which are accurate for both small and large variation (<i>V</i> close to two) are also obtained. In the special case of the information divergence they imply that <i>D</i> ≥ log[ 2/( 2-<i>V</i>)] - [( 2-<i>V</i>)/2] log[( 2+<i>V</i>)/2], which uniformly improves Vajda's inequality.

Subfields: Applied Mathematics, Mathematical Physics, Numerical Analysis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Mathematics

1. Igal Sason
2. Sergio Verdú
3. Tim van Erven
4. Peter Harremoës
5. Gennadiy P. Chistyakov
6. Friedrich Götze
7. Yingzhen Li
8. Ilya Mironov

In Mathematical Physics

1. Gennadiy P. Chistyakov
2. Friedrich Götze
3. Sergio Verdú
4. Tim van Erven
5. Igal Sason
6. Peter Harremoës
7. Roland Speicher
8. Serban Teodor Belinschi

In Numerical Analysis

1. Sergio Verdú
2. Tim van Erven
3. Igal Sason
4. Peter Harremoës

Citations in the corpus

• Theoretical evidence for adversarial robustness through randomization: the case of the Exponential family (2019, neutral): “Thanks to [20], one has dKL(μ1, μ2) ≥ 2dTV (μ1, μ2)(2) + 4dTV (μ1, μ2) 4 9 .”
• Rényi Divergence and Kullback-Leibler Divergence (2012, methodology): “In addition, Theorem 30 can be used to prove Gilardoni’s extension of Pinsker’s inequality from the case α = 1 to any α ∈ (0, 1] [25], which was mentioned in the introduction.”
• $f$ -Divergence Inequalities (2015, background): “(37) Remark 1: Different functions may lead to the same f -divergence for all (P,Q): if for an arbitrary b ∈ R, we have fb(t) = f0(t) + b (t− 1), t ≥ 0 (38) then Df0(P‖Q) = Dfb(P‖Q).”
• Rényi divergence and the central limit theorem (2016, background): “ Dα(P||Q) is a continuous function of the tuple (P,Q) with respect to the total variation distance in both coordinates. Conversely, it majorizes the total variation distance between Pand Q. Gilardoni [G] has shown that Dα(P||Q) ≥ α 2 kP−Qk2 TV. This extends the classical Pinsker inequality for the K”