On Divergences and Informations in Statistics and Information Theory

By Friedrich Liese, Igor Vajda. 2006

664 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Statistical and Nonlinear Physics, Statistics and Probability, Probabilistic & Bayesian Methods, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Statistical and Nonlinear Physics

In Statistics and Probability

In Probabilistic & Bayesian Methods

In Information Theory

Citations in the corpus

Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “(13) For the existence of the limits in these equations, where f(0) and f∗(0) can take values in R ∪ {∞}, as well as for the motivation for defining f ′(∞) .= limt→0 t f( 1t ), one may refer to (Liese & Vajda, 2006) and (Hiriart-Urruty & Lemaréchal, 2013, §2.3).”
f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization (2016, background): “A large class of different divergences are the so called f -divergences [5, 21], also known as the Ali-Silvey distances [1].”
Hilbert Space Embeddings and Metrics on Probability Measures (2009, background): “The concept of distance between probability measures is a fundamental one and has found many applications in probability theory, information theory and statistics (Rachev, 1991; Rachev and Rüschendorf, 1998; Liese and Vajda, 2006).”
Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk Minimization (2008, background): “See, for instance, [19] and references therein for a treatment of -divergences from this viewpoint.”
Imitation Learning as f-Divergence Minimization (2019, background): “We consider a broad class of such divergences called f-divergences [51, 52].”
The Right Complexity Measure in Locally Private Estimation: It is not the Fisher Information (2018, methodology): “…distributions The goal is then to provide upper bounds on the f -divergence D f ( M 0 || M 1 ) in terms of the channel Q ; the standard data-processing inequality [13, 37] [12, 14]), Thus, as long as the Dobrushin coeﬃcient is strictly positive, one obtains a strong data processing inequality.”
Strong Data Processing Inequalities and $\Phi $ -Sobolev Inequalities for Discrete Channels (2014, methodology): “(V.8) if we replace with its Csiszár conjugate (cid:2) ( u ) (cid:2) u (cid:2)( 1 / u ) [5].”
Mathematical Aspects of Divergence Based Vector Quantization Using Fr´ echet-Derivatives (2009, background): “Thus, as a famous example the Hellinger divergence [ 38 ],[55]:”
Privacy and Statistical Risk: Formalisms and Minimax Bounds (2014, methodology): “…For the special case that P ν,i = P ν for all i = 1 , . . . , n (for each ν = 0 , 1 ), this proves that The inequality M n 0 − M n 1 TV ≤ P n 0 − P n 1 TV is immediate from the classical data processing inequality (cf. [24, Theorem 14]), so proving inequality (26) is suﬃcient to prove the theorem.”
Hypothesis Testing Interpretations and Renyi Differential Privacy (2019, methodology): “We can extend this result to general measurable setting by using the continuity of ∆ ε (see also [Liese and Vajda, 2006]), which is obtained by f -divergence characterization of ∆ ε [Barthe and Olmedo, 2013].”
Limit theorems for eigenvectors of the normalized Laplacian for random graphs (2016, background): “The Chernoff information is a supremum of the Chernoff α-divergences for α ∈ (0, 1) and characterizes the error rate of the Bayes decision rule in hypothesis testing; the Chernoff α-divergence is an example of a f -divergence [1, 16] and it satisfies the information processing lemma and is invariant”
On the Validity of the Bootstrap in Non‐Parametric Functional Regression (2009, background): “For each fixed curve χ∈C20 at which the regression operator is evaluated, we indicate the particular f -divergence (for more details, see Lise & Vajda, 2006) called variational distance (in a 0–1 scale) between the true and the bootstrapped densities (i.e. b,h(χ)= ∫ | ftrue,χ(t) − fboot,χ(t)|dt).”
Statistical Inference on Random Dot Product Graphs: a Survey (2017, background): “Recall that any f -divergence satisfies the Information Processing Lemma and is invariant with respect to invertible transformations [60].”
Convexity/concavity of renyi entropy and α-mutual information (2015, background): “Tsallis divergence of order β was introduced in [8] and it is also known as the Hellinger divergence of order β [9].”