$f$ -Divergence Inequalities
By Igal Sason, Sergio Verdú. 2015
428 citations recorded by the source corpus.
This paper develops systematic approaches to obtain f -divergence inequalities, dealing with pairs of probability measures defined on arbitrary alphabets. Functional domination is one such approach, where special emphasis is placed on finding the best possible constant upper bounding a ratio of f -divergences. Another approach used for the derivation of bounds among f -divergences relies on moment inequalities and the logarithmic-convexity property, which results in tight bounds on the relative entropy and Bhattacharyya distance in terms of χ2 divergences. A rich variety of bounds are shown to hold under boundedness assumptions on the relative information. Special attention is devoted to the total variation distance and its relation to the relative information and relative entropy, including “reverse Pinsker inequalities,” as well as on the Eγ divergence, which generalizes the total variation distance. Pinsker's inequality is extended for this type of f -divergence, a result which leads to an inequality linking the relative entropy and relative information spectrum. Integral expressions of the Rényi divergence in terms of the relative information spectrum are derived, leading to bounds on the Rényi divergence in terms of either the variational distance or relative entropy.
Subfields: Applied Mathematics, Computational Mechanics, Statistical and Nonlinear Physics, Information Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Applied Mathematics
- L. Varshney
- Flávio du Pin Calmon
- Shahab Asoodeh
- Hsiang Hsu
- Richard Socher
- Nitish Shirish Keskar
- Maxim Raginsky
- Zhuowen Tu
In Computational Mechanics
- Flávio du Pin Calmon
- L. Varshney
- Shahab Asoodeh
- Hsiang Hsu
- Richard Socher
- Nitish Shirish Keskar
- Maxim Raginsky
- Dacheng Tao
In Statistical and Nonlinear Physics
- L. Varshney
- Flávio du Pin Calmon
- Shahab Asoodeh
- Hsiang Hsu
- Richard Socher
- Nitish Shirish Keskar
- Maxim Raginsky
- Zhuowen Tu
In Information Theory
- L. Varshney
- Flávio du Pin Calmon
- Shahab Asoodeh
- Hsiang Hsu
- Richard Socher
- Nitish Shirish Keskar
- Maxim Raginsky
- Zhuowen Tu
Citations in the corpus
- Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, neutral): “Then, we have With algebraic manipulation, we obtain, Replacing G (cid:48) in the RHS of the equation (E.2) by G (cid:48) = G + G (cid:48) ∆ G , and doing this recursively gives Corollary E.7.”
- Wasserstein Introspective Neural Networks (2017, methodology): “Based on the Pinsker’s inequality [37, 41], it is observed that”
- Strong Data Processing Inequalities and $\Phi $ -Sobolev Inequalities for Discrete Channels (2014, background): “More examples of Φ-divergences, as well as a wide variety of inequalities between them, can be found in [36].”
- On f-Divergences: Integral Representations, Local Behavior, and Inequalities (2018, background): “plified in the proof of Theorem 2. Theorem 1 yields integral representations for various f-divergences and related measures; some of these representations were previously derived by Sason and Verdu in [59] in a case by case basis, without the unified´ approach of Theorem 1. We next provide such integr”
- f-Divergence constrained policy improvement (2017, background): “The f-divergence (Csiszár, 1963; Morimoto, 1963; Ali and Silvey, 1966) generalizes many similarity measures between probability distributions (Sason and Verdu, 2016).”
- The total variation distance between high-dimensional Gaussians (2018, background): “For a survey on measures of distance between distributions and inequalities between them, see [9].”
- Limits of Detecting Text Generated by Large-Scale Language Models (2020, methodology): “We conjecture that one can indeed convert the reverse Pinsker inequality [33]:”
- A Better Bound Gives a Hundred Rounds: Enhanced Privacy Guarantees via f-Divergences (2020, neutral): “, [26].”
- A Survey on Knowledge Distillation of Large Language Models (2024, background): “Both Agarwal et al. (2024) and Sason and Verd´u (2016) assess the efficacy of different divergence functions in LLM distillation, finding the optimal divergence to be task-dependent.”
- Semiparametric Counterfactual Density Estimation (2021, background): “In contrast to typical f -divergences [Ali and Silvey, 1966, Csiszár, 1967, Rényi et al., 1961, Sason and Verdú, 2016], we allow the function f to have two arguments, one for each distribution; this allows us to capture not only f -divergences but also other distances such as those based on Lp…”