Rényi Divergence and Kullback-Leibler Divergence

By Tim van Erven, Peter Harremoës. 2012

1,563 citations recorded by the source corpus.

Rényi divergence is related to Rényi entropy much like Kullback-Leibler divergence is related to Shannon's entropy, and comes up in many settings. It was introduced by Rényi as a measure of information that satisfies almost the same axioms as Kullback-Leibler divergence, and depends on a parameter that is called its order. In particular, the Rényi divergence of order 1 equals the Kullback-Leibler divergence. We review and extend the most important properties of Rényi divergence and Kullback- Leibler divergence, including convexity, continuity, limits of σ-algebras, and the relation of the special order 0 to the Gaussian dichotomy and contiguity. We also show how to generalize the Pythagorean inequality to orders different from 1, and we extend the known equivalence between channel capacity and minimax redundancy to continuous channel inputs (for all orders) and present several other minimax results.

Subfields: Applied Mathematics, Electrical and Electronic Engineering, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Mathematics

  1. John C. Duchi
  2. Hongseok Namkoong
  3. Nishant A. Mehta
  4. Peter D. Grünwald
  5. Lawrence Carin
  6. Corinna Cortes
  7. Thomas Steinke
  8. Richard E. Turner

In Electrical and Electronic Engineering

  1. Kunal Talwar
  2. Thomas Steinke
  3. Mark Bun
  4. Vitaly Feldman
  5. Shiva Prasad Kasiviswanathan
  6. Yu-Xiang Wang
  7. John C. Duchi
  8. Li Zhang

In Statistical and Nonlinear Physics

  1. Wouter M. Kouw
  2. John C. Duchi
  3. Hongseok Namkoong
  4. Nishant A. Mehta
  5. Sylvain Gelly
  6. Mario Lucic
  7. Yingzhen Li
  8. Peter D. Grünwald

In Probabilistic & Bayesian Methods

  1. Yingzhen Li
  2. Yarin Gal
  3. Richard E. Turner
  4. Thijs van Ommen
  5. Nishant A. Mehta
  6. Sylvain Gelly
  7. Nicolas Papernot
  8. Lawrence Carin

In Information Theory

  1. Ningshan Zhang
  2. Vitaly Feldman
  3. Mark Bun
  4. Ananda Theertha Suresh
  5. Yu-Xiang Wang
  6. Yingzhen Li
  7. Nishant A. Mehta
  8. Peter D. Grünwald

Citations in the corpus