Information Theory and Statistics

By C. Craig, S. Kullback. 1960

Subfields: Statistics and Probability, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

In Information Theory

Citations in the corpus

On the Complexity of Bandit and Derivative-Free Stochastic Convex Optimization (2012, methodology): “To bound the divergence, we use the following standard result on the KL divergence between two Gaussians Kullback (1959): Lemma 5 Let N (µ, σ2) represent a Gaussian distribution variable with mean µ and variance σ2.”
Rényi Divergence and Kullback-Leibler Divergence (2012, background): “ion of D(P kP 0) in 0at in the interior of yields lim 0! 1 ( 0)2 D(P kP 0) = 1 2 J(); (49) where J() = E ( d d lnp ) 2 denotes the Fisher information at (see e.g. [30, Problem 12.7] or [48]). Haussler and Opper [6] argue that this property generalizes to lim 0! 1 ( 0)2 (P kP 0) = 2 J() (50) for any ”
Protein sectors: evolutionary units of three-dimensional structure (2009, methodology): “Minimum Discriminatory Information Method The minimum discriminatory information ( Kullback, 1997 ) (MDI) method gener-”
Approximating the Kullback Leibler Divergence Between Gaussian Mixture Models (2007, methodology): “Index Terms— Kullback Leibler divergence, variational methods, gaussian mixture models, unscented transformation.”
Effects of livestock breed and grazing intensity on grazing systems: 3. Effects on diversity of vegetation (2007, methodology): “The mutual information is proportional to the chi-square of a table (Kullback, 1959) and inversely correlated to Shannon’s entropy; the heterogeneity of a frequency table increases with a few types having a high frequency, and decreases with all types tending to have an even frequency.”
Efficient Learning in Boltzmann Machines Using Linear Response Theory (1998, methodology): “A suitable measure for the difference between the distributions pα and qα is the Kullback divergence (Kullback, 1959), K = ∑ α qα log qα pα .”
Outcomes of the International Union of Crystallography Commission on Powder Diffraction Round Robin on Quantitative Phase Analysis: samples 1a to 1h (2001, methodology): “The conventional KLD equation ( Kullback, 1968 ) has been weighted to allow for the concentration of the phase and is defined as”
A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress (2018, background): “stribution (Figure 8). Relative entropy IRL. Entropy optimization is also useful where the transition dynamics are not known. In reirl [28], the relative entropy ( known as KullbachLeibler divergence [29]) between two distributions on trajectories is minimized and the corresponding reward function i”
THE FORMAL DEFINITION OF REFERENCE PRIORS (2009, unknown): “The properties of f~ pjpg have been extensively studied; pioneering works include Gibbs [22], Shannon [38], Good [24, 25], Kullback and Leibler [35], Cherno [15], Jaynes [29, 30], Kullback [ 34 ] and Csiszar [18, 19].”
Entropy-Based Segregation Indices (2011, background): “Pp From a statistical perspective, Kullback (1959, page 158) shows that this”
ON CONVEX SOBOLEV INEQUALITIES AND THE RATE OF CONVERGENCE TO EQUILIBRIUM FOR FOKKER-PLANCK TYPE EQUATIONS (2001, unknown): “The well-known Csisz´ ar-Kullback inequality ([Csi63], [ Kul59 ]) shows that the logarithmic relative entropy (1.7) e = eχph is a ‘measure’ for the distance between two normalized L1 +(IR n )-functions ρ1 ,ρ 2 with IRn ρ1dx = IRn ρ2dx =1 :”