Information Theory And The Central Limit Theorem

By Oliver Johnson. 2004

323 citations recorded by the source corpus.

Subfields: Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

Citations in the corpus

A Kernelized Stein Discrepancy for Goodness-of-fit Tests (2016, background): “Fisher divergence, also known as Fisher information distance (Johnson, 2004), is deﬁned as that is, it is the L 2 ( p ) norm of s q ( x ) − s p ( x ) .”
Density Estimation in Infinite Dimensional Exponential Families (2013, neutral): “13 in [23]) between two continuously differentiable densities, p and q on an open set Ω ⊂ R, given as J(p‖q) = 1 2 ∫ p(x) ∥ ∥ ∥ ∥ ∂ log p(x) ∂x − ∂ log q(x) ∂x ∥ ∥ ∥ ∥ 2”
Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, methodology): “Fisher Divergence In order to avoid issues with computing the normalization term, we consider the Fisher divergence, or Fisher information distance (Johnson, 2004): As one can see from the formulation, in order to compute the Fisher divergence between two distributions, we need only have sampling…”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, unknown): “gence [48] DF (πθ∥πβ) to constrain the entropy-regularized learned policy.”
The generalized von Mises distribution (2007, unknown): “[7].”
Mechanisms in macroecology: AWOL or purloined letter? Towards a pragmatic view of mechanism (2010, methodology): “More profoundly, MaxEnt can be used to prove both the weak and strong versions of the CLT with reference to sums of random variables (Johnson 2004).”
Berry–Esseen bounds in the entropic central limit theorem (2011, unknown): “also [16].”
Stein's method for comparison of univariate distributions (2014, unknown): “[49, 61]).”
Stein’s density approach and information inequalities (2012, background): “, [20, 22, 14]), to the best of our knowledge the only references to explore this connection upfront are [5] in the context of compound Poisson approximation, and more recently [32, 33] for Poisson and Bernoulli approximation.”
A nonparametric method for the measurement of size diversity with emphasis on data standardization (2008, background): “Some probability distributions have known diversity expressions (e.g., Johnson 2004) and, in practice, some of these probability distributions fit well size data (Vidondo et al. 1997).”
Learning Energy-Based Models in High-Dimensional Spaces with Multiscale Denoising-Score Matching (2019, background): “…( ˜ xxx ) = ∇ ˜ xxx log ( p σ 0 ( ˜ xxx )) , and the corresponding objective is L SM is also known as the Fisher divergence or the Fisher information distance [21,22], L SM = D FD ( p σ 0 || p m ) , where p m ( xxx ) = e − E ( xxx ) / Z is the normalized distribution from the model energy function.”
Log-concavity and the maximum entropy property of the Poisson distribution (2006, background): “equality), − h(p)+h(φµ,σ2) = Z p(x)logp(x)dx− Z p(x)logφµ,σ2(x)dx = D(pkφµ,σ2) ≥ 0. (2) Such maximum entropy results can be regarded as the ﬁrst stage in proving convergence in relative entropy – see [8] for more such results. In Example 1.1, it is clear that the class of variables with mean µ and v”
Gradient Estimators for Implicit Models (2017, methodology): “al solutions in the appendix. In experiments we use Vstatistic solutions and leave comparisons between these methods to future work. The score matching objective (5) is also called Fisher divergence (Johnson, 2004) which is a special case of KSD by selecting K(x;x0) = x=x0. Thus the Stein gradient ”
Rényi divergence and the central limit theorem (2016, background): “uch strong CLT’s, including the non-i.i.d. situation and the problem of rates or Berry-Esseen bounds, were studied by many authors, and we refer an interested reader to [Li], [S-M], [A-B-B-N], [B-J], [J], [B-C-G2-4], [B-C-K], [B-C]. As for convergence in Dα with α>1, not much is known so far. Thi”
Efficient Learning of Generative Models via Finite-Difference Score Matching (2020, methodology): “Score matching (SM) [21] is an alternative objective that circumvents the intractable partition function by training unnormalized models with the Fisher divergence [23], which depends on the Hessian trace and (Stein) score function [37] of the log-density function.”
Rate of convergence and Edgeworth-type expansion in the entropic central limit theorem (2011, background): “This has been conjectured by Johnson [14], page 49.”