Information-theoretic upper and lower bounds for statistical estimation

By Tong Zhang. 2006

212 citations recorded by the source corpus.

Subfields: Computational Mechanics, Computer Networks and Communications, Electrical and Electronic Engineering, Statistical Learning Theory, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Networks and Communications

In Electrical and Electronic Engineering

In Statistical Learning Theory

In Information Theory

Citations in the corpus

PAC-Bayesian Theory Meets Bayesian Inference (2016, neutral): “…we can relate the empirical loss (cid:98) L ,Y of a predictor to its likelihood: or, the other way around, Unfortunately, existing PAC-Bayesian theorems work with bounded loss functions or in very speciﬁc contexts [ e.g. , 10, 49], and (cid:96) nll spans the whole real axis in its general form.”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, methodology): “t goes by the names Safe Bayes, generalized Bayesian inference, and fractional Bayesian inference (e.g., de Heide et al. ,2019 ;Gr¨unwald et al. 2017 Barron & Cover ,1991 ;Walker & Hjort 2001 Zhang 2006 Bissiri et al.,2016;Grunwald¨ ,2012). Safe Bayes has been shown to be natural from a vari”
Adaptive Bayesian density estimation with location-scale mixtures (2010, result): “The determination of lower bounds for convergence rates deserves further investigation; some results can be found in [33].”
User-friendly introduction to PAC-Bayes bounds (2021, methodology): “However, in the same way PAC-Bayes community not aware of Zhang’s “information theoretic bounds” [206], it seems that the authors of [200] are not aware of PAC-Bayes bounds.”
Fast Rates with Unbounded Losses (2016, methodology): “The special case of this condition with η = 1 under log loss has appeared previously, often implicitly, in works studying rates of convergence in density estimation (Barron and Cover, 1991; Li, 1999; Zhang, 2006a; Kleijn and van der Vaart, 2006; Grünwald, 2011).”
The Safe Bayesian - Learning the Learning Rate via the Mixability Gap (2012, methodology): “In relative PAC-Bayesian bounds (Audibert, 2004, Zhang, 2006b, Catoni, 2007) one proves frequentist convergence bounds of randomized predictors which depend on some user-specified “prior” distribution over Θ.”
Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It (2014, methodology): “…hold, as the authors emphasize, under regularity conditions that are substantially stronger than those needed for consistency when the model is correct (as by e.g. Ghosal et al. (2000) or Zhang (2006a)), and our example shows that consistency may fail to hold even in relatively simple problems.”
On Bayes Risk Lower Bounds (2014, methodology): “P) ˙ : (6) A special case of the above inequality has appeared previously inZhang(2006, Theorem 6.1) (please refer to Remark4.5for a detailed explanation of the connection between inequality (6) and (Zhang, 2006, Theorem 6.1)). We also prove analogues of the above inequality for dierent fdivergences”
Assigning a value to a power likelihood in a general Bayesian model (2017, background): “See, for example, Royall & Tsou (2003), Zhang (2006a), Zhang (2006b), Jiang & Tanner (2008), Bissiri, Holmes & Walker (2016), Walker & Hjort (2001), Watson & Holmes (2016), Miller & Dunson (2015), Grünwald & van Ommen (2014), Syring & Martin (2015), and more generally Hansen & Sargent (2008).”
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy (2017, background): “Similar connections have been made by Zhang (2006a), Zhang (2006b), Grünwald (2012), and Grünwald and Mehta (2016).”
Follow the leader if you can, hedge if you must (2013, methodology): “Such learning rates pop up in batch settings as diverse as L1/L2-regularized regression such as Lasso and Ridge, standard Bayesian nonparametric and PAC-Bayesian inference (Zhang, 2006; Audibert, 2004; Catoni, 2007), and — as in this paper — in sequential prediction.”
Approximate Models and Robust Decisions (2014, methodology): “This has the form of a Gibbs posterior or an exponentially weighted PAC-Bayesian approach (Zhang, 2006a, 2006b, Bissiri, Holmes and Walker, 2013, Dalalyan and Tsybakov, 2008, 2012).”
The power of localization for efficiently learning linear separators with noise (2013, methodology): “agreement based active learning literature [Balcan et al. 2006; Hanneke 2007; Koltchinskii 2010; Hanneke 2011; Wang 2011] and earlier in passive learning [Bartlett et al. 2005; Boucheron et al. 2005; Zhang 2006]. Journal of the ACM, Vol. V, No. N, Article A, Publication date: January YYYY. Learning ”
Information-theoretic analysis of generalization capability of learning algorithms (2017, methodology): “To solve the relaxed optimization problem in (23), ﬁrst note that inf It follows that for each s ∈ Z n , the algorithm P ∗ W | S that minimizes (23) satisﬁes The solution to (D.12) for each s ∈ Z n turns out to be the Gibbs algorithm [19] as described in (24), which does not depend on µ .”
Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity (2008, methodology): “� = {1,...,M }, and the distribution � is uniform on �. Procedures of the type (2)–(3) with general sets � and priors � came into consideration quite recently [9, 10, 36, 4, 41, 42 , 1, 2], partly in connection with the PAC-Bayesian approach.”