Rényi Divergence and Kullback-Leibler Divergence
By Tim van Erven, Peter Harremoës. 2012
1,563 citations recorded by the source corpus.
Rényi divergence is related to Rényi entropy much like Kullback-Leibler divergence is related to Shannon's entropy, and comes up in many settings. It was introduced by Rényi as a measure of information that satisfies almost the same axioms as Kullback-Leibler divergence, and depends on a parameter that is called its order. In particular, the Rényi divergence of order 1 equals the Kullback-Leibler divergence. We review and extend the most important properties of Rényi divergence and Kullback- Leibler divergence, including convexity, continuity, limits of σ-algebras, and the relation of the special order 0 to the Gaussian dichotomy and contiguity. We also show how to generalize the Pythagorean inequality to orders different from 1, and we extend the known equivalence between channel capacity and minimax redundancy to continuous channel inputs (for all orders) and present several other minimax results.
Subfields: Applied Mathematics, Electrical and Electronic Engineering, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods, Information Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Applied Mathematics
- John C. Duchi
- Hongseok Namkoong
- Nishant A. Mehta
- Peter D. Grünwald
- Lawrence Carin
- Corinna Cortes
- Thomas Steinke
- Richard E. Turner
In Electrical and Electronic Engineering
- Kunal Talwar
- Thomas Steinke
- Mark Bun
- Vitaly Feldman
- Shiva Prasad Kasiviswanathan
- Yu-Xiang Wang
- John C. Duchi
- Li Zhang
In Statistical and Nonlinear Physics
- Wouter M. Kouw
- John C. Duchi
- Hongseok Namkoong
- Nishant A. Mehta
- Sylvain Gelly
- Mario Lucic
- Yingzhen Li
- Peter D. Grünwald
In Probabilistic & Bayesian Methods
- Yingzhen Li
- Yarin Gal
- Richard E. Turner
- Thijs van Ommen
- Nishant A. Mehta
- Sylvain Gelly
- Nicolas Papernot
- Lawrence Carin
In Information Theory
- Ningshan Zhang
- Vitaly Feldman
- Mark Bun
- Ananda Theertha Suresh
- Yu-Xiang Wang
- Yingzhen Li
- Nishant A. Mehta
- Peter D. Grünwald
Citations in the corpus
- Multiple-Source Adaptation with Domain Classifiers (2020, background): “We will denote by d α ( D D ′ ) the exponential: The following lemma from Van Erven and Harremos [2014] summarizes some useful properties of the Rényi divergence.”
- Rényi Differential Privacy of the Sampled Gaussian Mechanism (2019, methodology): “ p TN(f(T);˙2Id); where the sum here denotes mixing of the distributions with the weights p T. Similarly, M(S0) = X T p T (1 q)N(f(T);˙2Id)+qN(f(T[fxg);˙2Id) : 3 R´enyi divergence is quasi-convex [vEH14], allowing us to bound D M(S) M(S0) sup T D N(f(T);˙2Id) (1 q)N(f(T);˙2Id)+qN(f(T[fxg);˙2Id) ”
- Privacy Amplification by Iteration (2018, neutral): “Proposition 5 ( [30]).”
- Subsampled Rényi Differential Privacy and Analytical Moments Accountant (2018, methodology): “…allows one to find the smallest (cid:15) given a desired δ or vice versa using: Using the convexity of CGF K M ( λ ) and monotonicity of K M ( λ ) /λ in λ (Van Erven & Harremos, 2014, Corollary 2, Theorem 3), we observe that the optimization problem in (4) is log-convex and the optimization…”
- Rényi Differential Privacy (2017, background): “The following statement is a closed-form expression of the Rényi divergence between a Gaussian and its offset (for a more general version see [19], [21]).”
- Scalable Private Learning with PATE (2018, positive): “The Rényi divergence between two normal distributions can be computed in closed form (van Erven & Harremoës, 2014): Dλ(P‖Q) = λ (f(D)− f(D′))2 2σ2s2 + 1 1− λ ln s SSβ(D)1−λ · SSβ(D′)λ , (11) provided s2 , (1− λ) · SSβ(D)2 + λ · SSβ(D′)2 > 0.”
- Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds (2016, background): “ Q: e (1)D PkQ) = E Z˘PrivLoss(PkQ) e( 1)Z for all 2(1;1). Moreover, D 1 (PkQ) = E Z˘PrivLoss(PkQ) [Z]. We record several useful and well-known properties of Renyi divergence. We refer the reader to [vEH14] for proofs and discussion of these (and many other) properties. Selfcontained proofs are give”
- The Perception-Distortion Tradeoff (2017, background): “KL, TV, Hellinger, X (2)) as well as the Renyi divergence, satisfy this assumption [5, 38].”
- Dropout Inference in Bayesian Neural Networks with Alpha-divergences (2017, positive): “When α/N → 0, the unnormalised density in (6) converges to q̃(ω) for every ω, and Zq → 1 by the assumption of Zq +∞ (Van Erven & Harremoës, 2014).”
- Calibrating Noise to Variance in Adaptive Data Analysis (2017, methodology): “the bound we will use: Corollary 2.7. Let Xand Y be real-valued random variables and t>0. Then E[ X] ≤ 1 t D( kY) +lnE h etY i . Next we note that KL divergence is a convex function. Lemma 2.8 ([VEH14], Theorem 11). Let P0, P1, Q0, Q1 be probability distributions on the same space Ω. For t∈ (0,1”
- A Review of Domain Adaptation without Target Labels (2019, neutral): “Another example is the Rényi divergence [78]:”
- Learning Models with Uniform Performance via Distributionally Robust Optimization (2018, background): “The Rényi α-divergence [98] between distributions P and Q is”
- Fast Rates with Unbounded Losses (2016, background): “Now, from Lemma 2 of van Erven and Harremoës (2014), for y ≥ 12 we have (y − 1) (1 + 1 − y 2 ) ≤ log y ≤ y − 1.”
- Federated Learning with Bayesian Differential Privacy (2019, background): “Van Erven and Harremos [31] provide a good survey of R ´ enyi divergence properties in general.”