On Information and Sufficiency

By Huaiyu Zhu. 1997

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Theory

Citations in the corpus

Variational Inference: A Review for Statisticians (2016, background): “(14) The bound then follows from the fact that kl ( · ) ≥ 0 (Kullback and Leibler 1951).”
Deep learning in neural networks: An overview (2014, methodology): “Many UL methods are designed to maximize entropy-related, information-theoretic (Boltzmann, 1909; Kullback & Leibler, 1951; Shannon, 1948) objectives (e.g., Amari, Cichocki, & Yang, 1996; Barlowet al., 1989;Dayan&Zemel, 1995;Deco&Parra, 1997; Field, 1994; Hinton, Dayan, Frey, & Neal, 1995; Linsker,…”
Relational autoencoder for feature extraction (2017, methodology): “It makes a strong assumption concerning the distribution of latent neurons and tries to minimize the difference between a posterior distribution and the distribution of latent neurons with difference measured by the Kullback-Leibler divergence [44].”
Few-Shot Adversarial Domain Adaptation (2017, background): “Traditional unsupervised DA (UDA) paradigms try to align the distributions of the features in the feature space, mapped from the source and the target domains using a metric between distributions, Maximum Mean Discrepancy [20] being a popular one and other metrics like Kullback Leibler divergence [2”
Meta-Learning: A Survey (2018, neutral): “Another way to deﬁne task similarity is to take the existing evaluations P i,j , use Thompson Sampling [166] to obtain the optima distribution ρ jmax , and then measure the KL-divergence [80] between ρ jmax and ρ newmax [123].”
An Algorithmic Perspective on Imitation Learning (2018, methodology): “In the field of information geometry, Kullback-Leibler (KL) divergence is used to quantify the difference between two probability distributions [Kullback and Leibler, 1951] DKL (p(x)||q(x)) = ∫ p(x) ln p(x) q(x) dx. (3.19) Since the KL divergence measures the difference between two probability…”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “”2 While FNNs are traditionally linked [23] to concepts of statistical mechanics and information theory [24, 257, 136], the programs of general computers such as RNNs call for the framework of Algorithmic Information Theory (AIT) [263, 130, 33, 145, 264, 147] (own AIT work: [234, 235, 236, 237, 238]”
A Comprehensive Survey on Transfer Learning (2019, neutral): “Measurement Related Algorithms Maximum Mean Discrepancy [28] [29] [30] [31] [32]· · · Kullback-Leibler Divergence [33] [34] [35] [36] [37]· · · Jensen-Shannon Divergence [38] [39] [40] [41] [42]· · · Bregman Divergence [43] [44] [45] [46] [47]· · · Hilbert-Schmidt Independence Criterion [48] [29] [4”
Entropy Search for Information-Efficient Global Optimization (2011, background): “A functional that is well-deﬁned on probability densities and preserves many of the information-content interpretations of entropy (Jaynes and Bretthorst, 2003) is relative entropy , also known as Kullback-Leibler divergence (Kullback and Leibler, 1951).”
Filters, Random Fields and Maximum Entropy (FRAME): Towards a Unified Theory for Texture Modeling (1998, positive): “(18)), since our goal is to estimate the underlying distribution f (I), the goodness of p(I ;3K , SK ) can be measured by the Kullback-Leibler distance between p(I ;3K , SK ) and f (I) (Kullback and Leibler, 1951):”
Community detection in graphs (2009, methodology): “Here the snapshot cost28 is the Kullback-Leibler (KL) divergence (Kullback and Leibler, 1951) between the adjacency/similarity matrix at time t and the matrix describing the community structure of the graph at time t; the history cost is the KL divergence between the matrices describing the…”
A Survey on Metric Learning for Feature Vectors and Structured Data (2013, methodology): “It generalizes many widely-used measures: the Mahalanobis distance is recovered by setting ϕ(x) = 12x TMx, the KL divergence (Kullback and Leibler, 1951) by choosing ϕ(p) = ∑d i=1 pi log pi (here, p is a discrete probability distribution), etc. Wu et al. consider the following symmetrized…”
Multi-agent Diverse Generative Adversarial Networks (2017, positive): “We use KL-divergence [18] and number of modes recovered [7] as the criterion for comparison and show superior results compared to all other methods.”
Potentials of Mean Force for Protein Structure Prediction Vindicated, Formalized and Generalized (2010, result): “This qualitative result is confirmed by the Kullback-Leibler divergence [50] – a natural distance measure for probability distributions expressed in bits – between the target distribution and the resulting marginal distributions of rg.”
Reciprocal processes. A measure-theoretical point of view (2013, background): “The new ingredient of this section is the relative entropy, or Kullback-Leibler divergence, introduced in [KL51].”