On Information and Sufficiency

By S. Kullback, R. A. Leibler. 1951

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Theory

Citations in the corpus

Stochastic variational inference (2012, methodology): “A Riemannian metric for the parameter of a probability distribution is the Fisher information (Amari, 1982; Kullback and Leibler, 1951), G(λ)=E[(∇λ log p(β |λ))(∇λ log p(β |λ))>] .”
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “Since the true posterior is typically intractable, an approximate distribution qθ(w|D) is defined that is aimed to be as similar as possible to the true posterior p(w|D), measured by the Kullback-Leibler (KL) divergence [35].”
A survey on data fusion in internet of things: Towards secure and privacy-preserving fusion (2019, background): “AC CE PT ED M AN US CR IP T divergence [71] was introduced to evaluate the importance of sensors to help select the most important sensors for data collection, which can help enhance the reliability and robustness.”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, background): “Leibler divergence [65] between the random variable defined by unit i and the one corresponding to the desired activations will measure how different both distributions are [66]:”
Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence (2018, positive): “To achieve this, we regularize over the conditional likelihood distributions pθ(y|x) using the approximate KL-divergence, Eq.”
On Choosing and Bounding Probability Metrics (2002, background): “(4) Relative entropy was ﬁrst deﬁned by Kullback and Leibler (1951) as a generalization of the entropy notion of Shannon (1948).”
Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning (2017, methodology): “…approaches for the choice of the distance function are the Proxy A-distance (Ben-David et al., 2010), the Kullback-Leibler (KL) divergence Kullback & Leibler (1951), applied to the mean of the activations (Zhuang et al., 2015), and the Maximum Mean Discrepancy (Gretton et al., 2006, MMD).”
The Akaike information criterion: Background, derivation, properties, application, interpretation, and refinements (2019, methodology): “…approximation is determined by employing a well-known measure that can be used in assessing the similarity between the generating model g(y) and a candidate model f yjθkð Þ: the Kullback–Leibler information (Kullback, 1968; Kullback & Leibler, 1951), also known as the Kullback–Leibler divergence.”
Causality detection based on information-theoretic approaches in time series analysis (2007, background): “The Kullback-Leibler divergence (KLD, also called relative entropy or crossentropy), introduced by Kullback and Leibler [137], is an alternative approach to mutual information.”
Pre-trained Summarization Distillation (2020, background): “Direct Knowledge Distillation (KD) In the KD setting, even more information is transferred from teacher to student, by encouraging the student to match the teacher’s full probability distribution over possible next words at each position, by minimizing KL-Divergence(Kullback and Leibler, 1951; Sanh ”
Predicting clicks: estimating the click-through rate for new ads (2007, methodology): “Our measure of performance is the average KL-divergence [13] between the model’s predicted CTR and the true CTR on the test set (lower is better).”
Natural Evolution Strategies (2008, background): “One such natural distance measure between two probability distributions is the Kullback-Leibler divergence (Kullback and Leibler, 1951).”
Identifying fake Amazon reviews as learning from crowds (2014, methodology): “frequents, in order to identify the features most effective in discriminating the two classes of reviews, the Information Gain (IG) of the selected ngrams was computed (Kullback and Leibler, 1951; Yang and Pedersen, 1997).”