Information Theory and Statistics

By E. Haroutunian. 2011

3,591 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Dropout Inference in Bayesian Neural Networks with Alpha-divergences (2017, background): “…distribution q θ ( ω ) (parametrised by variational parameters θ ), and then minimise w.r.t. θ the KL divergence (Kullback & Leibler, 1951; Kullback, 1959) between the approximating distribution and the full posterior: where A ∝ B is slightly abused here to denote equality up to an…”
Mutual Information Neural Estimation (2018, methodology): “In order to achieve a general-purpose estimator, we rely on the well-known characterization of the mutual information as the Kullback-Leibler (KL-) divergence (Kullback, 1997) between the joint distribution and the product of the marginals (i.e., I ( X ; Z ) = D KL ( P XZ || P X ⊗ P Z ) ).”
Graph-based Knowledge Distillation by Multi-head Attention Network (2019, methodology): “(9) deﬁnes the loss of the transfer task by applying Kullback-Leibler divergence (KLD) [15] to G T and G S which are obtained from the TN and SN, respectively.”
Inferring Biological Networks by Sparse Identification of Nonlinear Dynamics (2016, background): “For example, partnering the Kullback-Leibler (KL) divergence [10], [11], a measure of information loss between empirically collected data and model generated data, and the Akaike information criteria (AIC) [12], [13], a relative estimate of information loss across models balancing model complexity a”
Spatiotemporal Saliency in Dynamic Scenes (2010, neutral): “(KL) divergence between the probability distributions pXðxÞ and qXðxÞ [ 20 ].”
Anomaly Detection and Localization in Crowded Scenes (2014, positive): “…[13]: SðlÞ ¼ X 1 c¼0 fp CðlÞ ðcÞKL½p XjCðlÞ ðx x x xjcÞkp X ðx x x xÞg; ð4Þ where p XjCðlÞ ðx x x xjcÞ are class-conditional densities and KLðp q k Þ ¼ R X p X ðx x x xÞ log p X ðx x x xÞ q X ðx x x xÞ dx x x x the Kullback-Leibler (KL) divergence between p X ðx x x xÞ and q X ðx x x xÞ [30].”
Least-squares two-sample test (2011, methodology): “This problem is referred to as the two-sample test or the homogeneity test (Kullback, 1959).”
Generic and Scalable Framework for Automated Time-series Anomaly Detection (2015, methodology): “We have used Kernel Density Estimation [7] to non-parametrically estimate the distribution of the residuals and the KullbackLeibler divergence [16] to measure the change in the distribution.”
Of Bits and Wows: A Bayesian Theory of Surprise with Applications to Attention (2009, methodology): “In what follows, for standard well-known theoretical reasons such as invariance with respect to reparameterizations, we use the relative entropy or Kullback–Liebler (Kullback, 1968) where H denotes the entropy.”
Bidirectional Helmholtz Machines (2015, background): “Solomon Kullback and Richard Leibler in 1951 introduced KL divergence to probability theory as a measure of dissimilarity between two probability distributions [56, 54, 55].”
Information geometry and sufficient statistics (2012, background): “Note that similar regularity conditions have been posed by Cramer [12, p.500-501], see also [24, Chapter 2, § 6].”
Convex Formulations for Fair Principal Component Analysis (2018, methodology): “For the special case X+ ∼ N (μ+,Σ+) and X− ∼ N (μ−,Σ−), we have (Kullback 1997):”
Beyond Gaussian Statistical Modeling in Geophysical Data Assimilation (2010, methodology): “These tools will be reviewed in this section, and, moreover, some of them will be used in section 6. a. Relative entropy A measure of the discrepancy between two pdfs p and q is given by the Kullback–Leibler divergence or relative entropy (Kullback 1959): K(p, q) 5 ð dx p(x) ln p(x) q(x) .”
White-box vs Black-box: Bayes Optimal Strategies for Membership Inference (2019, background): “The inverse covariance matrix C − 1 is asymptotically n times the Fisher matrix (van der Vaart, 1998), which itself is the Hessian of the loss (Kullback, 1997): We denote by θ ∗ 0 (resp. θ ∗ 1 ) the mode of the Gaussian corresponding to { z 2 , ..., z n } (resp.”
A Global Human Settlement Layer From Optical HR/VHR RS Data: Concept and First Results (2013, background): “Given continuous measurements and their respective classes , their classification power is best captured by the Receiver Operational Characteristics (ROC) [71], which represents the probabilities of missed detection and false alarm for various classification thresholds:”