MUTUAL INFORMATION, METRIC ENTROPY AND CUMULATIVE RELATIVE ENTROPY RISK

By David Haussler, Manfred Opper. 1997

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Entropy-SGD: biasing gradient descent into wide valleys (2016, background): “Motivated by the same ﬁnal goal, viz. ﬂat local minima, the authors in Hochreiter & Schmidhuber (1997b) introduce hard constraints on the training loss and the width of local minima and show using the Gibbs formalism (Haussler et al., 1997) that this leads to improved generalization.”
Rényi Divergence and Kullback-Leibler Divergence (2012, background): “Index Terms—α-divergence, Bhattacharyya distance, information divergence, Kullback-Leibler divergence, Pythagorean inequality, Rényi divergence I. INTRODUCTION SHANNON entropy and Kullback-Leibler divergence (alsoknown as information divergence or relative entropy) are perhaps the two most fundamen”
Dataset Shift in Machine Learning (2009, background): “, we play the game of “guessing a probability distribution” [Haussler and Opper, 1997].”
The Minimum Description Length Principle in Coding and Modeling (1998, background): “For a recent treatment of asymptotics and metric entropy characterization of the latter quantity see Haussler and Opper [30].”
Upper and Lower Bounds on the Learning Curve for Gaussian Processes (2000, background): “Haussler and Opper (1997) have developed general (asymptotic) bounds for the expected log-likelihood of a test point after seeing n training points.”
$f$ -Divergence Inequalities (2015, background): “SUMMARY Since many distance measures of interest fall under the common paradigm of an f -divergence, it is not surprising that bounds on the ratios of various f -divergences are useful in many instances such as proving convergence of probability measures according to various metrics, analysis of rat”
Predictability , Complexity , and Learning (2002, methodology): “…vector of parameters ®, the literature on bounding similar quantities is especially rich (Haussler, Kearns, & Schapire, 1994; Wong & Shen, 1995; Haussler & Opper, 1995, 1997, and references therein), and related asymptotic calculations have been done (Clarke & Barron, 1990; MacKay, 1992;…”
A Bayesian Rule for Adaptive Control based on Causal Interventions (2009, background): “The distribution that minimizes the KL-divergence is a Bayesian mixture distribution over all possible environments [Haussler and Opper, 1997, Opper, 1998].”
Predictability, Complexity, and Learning (2000, methodology): “This is easily shown to be equal to the predictive information about all of the future, and it is also the “cumulative information gain” (Haussler et al. 1994) or the “cumulative relative entropy risk” ( Haussler and Opper 1997 ).”
Prediction, learning, and games (2006, methodology): “We mention the important work of Haussler and Opper [152].”
On Bayes Risk Lower Bounds (2014, methodology): “Tightest general upper bounds for the mutual information, to the best of our knowledge, are proved in Haussler and Opper (1997) and, in Section 5, we provide inequalities analogous to theirs for more general f -informativities.”
A Minimum Relative Entropy Principle for Learning and Acting (2008, background): “The set Z := A×O is the interaction set .”
On the measure of the information in a statistical experiment (2007, background): “It is also the Bayes risk when estimating the true density, pθ, under the KL-divergence loss (see, e.g., Haussler and Opper 1997).”
Catching up faster by switching sooner: a predictive approach to adaptive estimation with an application to the AIC–BIC dilemma (2012, background): “) Minimax cumulative risk has previously been studied by, among others, Haussler and Opper (1997), Rissanen et al.”
Mixing Strategies for Density Estimation (2000, background): “Density estimation is closely related to universal coding as illustrated in Barron (1987), Clark and Barron (1990), Barron and Cover (1991), Yang (1996) (a formal statement is given as Lemma 2.6) and Haussler and Opper (1997).”