From ɛ-entropy to KL-entropy: Analysis of minimum information complexity density estimation

By Tong Zhang. 2006

230 citations recorded by the source corpus.

We consider an extension of e-entropy to a KL-divergence based complexity measure for randomized density estimation methods. Based on this extension, we develop a general information-theoretical inequality that measures the statistical complexity of some deterministic and randomized density estimators. Consequences of the new inequality will be presented. In particular, we show that this technique can lead to improvements of some classical results concerning the convergence of minimum description length and Bayesian posterior distributions. Moreover, we are able to derive clean finite-sample convergence bounds that are not obtainable using previous approaches.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Statistics and Probability

Citations in the corpus

Fused Lasso Additive Model (2014, background): “…additive modeling in high dimensions has been an active area of research in recent years (Bühlmann and Yu 2003; Sardy and Tseng 2004; Lin and Zhang 2006; Avalos, Grandvalet, and Ambroise 2007; Bühlmann and Hothorn 2007; Huang, Horowitz, and Wei 2010; Zhang, Cheng, and Liu 2011; Wood,…”
Rényi Divergence and Kullback-Leibler Divergence (2012, methodology): “Rényi divergence appears as a crucial tool in proofs of convergence of minimum description length and Bayesian estimators, both in parametric and nonparametric models [5], [6], [7, Ch.”
Assigning a value to a power likelihood in a general Bayesian model (2017, background): “See, for example, Royall & Tsou (2003), Zhang (2006a), Zhang (2006b), Jiang & Tanner (2008), Bissiri, Holmes & Walker (2016), Walker & Hjort (2001), Watson & Holmes (2016), Miller & Dunson (2015), Grünwald & van Ommen (2014), Syring & Martin (2015), and more generally Hansen & Sargent (2008).”
Bayesian inference with rescaled Gaussian process priors (2007, unknown): “pled from a ﬁxed true distribution, a prior on an inﬁnite-dimensional model has to be carefully chosen in order to obtain optimal rates of contraction of the posterior (cf. e.g. [3], [17], [19], [4], [25], [23]). Even if posterior consistency is ensured, the rate of contraction of the posterior arou”
Dynamics of Bayesian Updating with Dependent Data and Misspecified Models (2009, methodology): “(37; 67; 5; 6) on mis-specified models do not hold because the data are dependent.”
Calibrating general posterior credible regions (2015, methodology): “This formulation is inspired by the literature on Gibbs posteriors, where data and parameter of interest are connected via a loss function, instead of a likelihood; see, e.g., Bissiri et al. (2016), Alquier et al. (2016), Zhang (2006), Jiang and Tanner (2008), and Syring and Martin (2017).”
Fast Rates with Unbounded Losses (2016, methodology): “The special case of this condition with η = 1 under log loss has appeared previously, often implicitly, in works studying rates of convergence in density estimation (Barron and Cover, 1991; Li, 1999; Zhang, 2006a; Kleijn and van der Vaart, 2006; Grünwald, 2011).”
Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It (2014, methodology): “…hold, as the authors emphasize, under regularity conditions that are substantially stronger than those needed for consistency when the model is correct (as by e.g. Ghosal et al. (2000) or Zhang (2006a)), and our example shows that consistency may fail to hold even in relatively simple problems.”
Asymptotically minimax empirical Bayes estimation of a sparse normal mean vector (2013, methodology): “The use of pseudo-posteriors, where an inverse temperature parameter plays the role of κ , has been considered in the statistics and machine learning literature (e.g., Dalalyan and Tsybakov 2008; Jiang and Tanner 2008; Zhang 2006), but our context is diﬀerent.”
Approximate Models and Robust Decisions (2014, methodology): “This has the form of a Gibbs posterior or an exponentially weighted PAC-Bayesian approach (Zhang, 2006a, 2006b, Bissiri, Holmes and Walker, 2013, Dalalyan and Tsybakov, 2008, 2012).”
Linear and convex aggregation of density estimators (2006, methodology): “The work on this subject is mainly devoted to the MS aggregation with the Kullback-Leibler divergence as a loss function ([6], [7], [29], [30]), and is based on information-theoretical ideas close to the earlier papers: [1] and [13].”
Follow the leader if you can, hedge if you must (2013, methodology): “Such learning rates pop up in batch settings as diverse as L1/L2-regularized regression such as Lasso and Ridge, standard Bayesian nonparametric and PAC-Bayesian inference (Zhang, 2006; Audibert, 2004; Catoni, 2007), and — as in this paper — in sequential prediction.”
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy (2017, background): “Similar connections have been made by Zhang (2006a), Zhang (2006b), Grünwald (2012), and Grünwald and Mehta (2016).”
Bayesian variable selection for high dimensional generalized linear models : Convergence rates of the fitted densities (2007, methodology): “o use here, through which we have established our results; there exist several other alternatives, possibly with more relaxed conditions, for example, in Ghosal, Ghosh and van der Vaart [10] or Zhang [26]. Suppose Pn is a sequence of sets of probability densities. (For each n, denote Pc nas the comp”