PAC-Bayesian model averaging

By David A. McAllester. 1999

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, methodology): “While the raw parameter count provides only a crude approximation of a model’s capacity, more sophisticated metrics such as those based on PAC-Bayes [McAllester, 1999, Dziugaite and Roy, 2017, Neyshabur et al., 2017b], VC dimension [Vapnik and Chervonenkis, 1971], and parameter norms [Bartlett et al”
Incorporating neuro-inspired adaptability for continual learning in artificial intelligence (2023, positive): “Formally, we analyse the generalization errors of a certain solution for continual learning with PAC-Bayes theory 69 , to provide the objective for computational modelling of neurological adaptive mechanisms.”
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, neutral): “Inspired by the PAC-Bayes theory [24] and previous work in domain generalization [5, 4], we present the upper bounds of these two errors as follows (proof in Appendix A):”
A PAC-Bayesian bound for Lifelong Learning (2013, background): “It is then possible to prove generalization bounds such as the following: with probability at least 1 − δ (over the sampling of S ) we have for all distributions Q over H (McAllester, 1999) where P is a reference or prior distribution over H that must be chosen before observing the samples S , and…”
Exploring Generalization in Deep Learning (2017, background): “The PAC-Bayesian framework [16, 17] provides guarantees on the expected error of a randomized predictor (hypothesis), drawn form a distribution denotedQ and sometimes referred to as a “posterior” (although it need not be the Bayesian posterior), that depends on the training data.”
A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks (2017, background): “The PAC-Bayesian framework [9, 10] provides generalization guarantees for randomized predictors, drawn form a learned distribution Q (as opposed to a learned single predictor) that depends on the training data.”
Stronger generalization bounds for deep nets via a compression approach (2018, methodology): “Using PAC-Bayes bound [McAllester, 1998, 1999] this noise stability yielded generalization bounds for fully connected nets of depth 2.”
Fantastic Generalization Measures and Where to Find Them (2019, methodology): “Sharpness-based measures like PAC-Bayesian bounds (McAllester, 1999) and the measure proposed by Keskar et al. (2016) perform best and seem promising candidates for further research.”
A PAC-Bayesian Tutorial with A Dropout Bound (2013, methodology): “The ﬁrst PAC-Bayesian theorem was given in [11].”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, background): “In general, PAC-Bayes provides a compelling framework for deriving explicit non-asymptotic generalization bounds [31, 23, 7, 36, 37, 30, 17].”
Stability and Generalization in Structured Prediction (2016, methodology): “…we further have that (Z1, . . . , Zn) are independent and identically distributed (i.i.d.)—for instance, if they represent “micro examples” drawn independently from some target distribution—then Corollary 1 reduces to standard PAC-Bayes bounds for learning from i.i.d. data (e.g., McAllester, 1999).”
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy (2017, methodology): “We now present a PAC-Bayes theorem, ﬁrst established by McAllester (1999).”
Statistical learning theory and stochastic optimization (2004, methodology): “It is also an active field of research in data compression theory [82] and in computer science probabilistic learning theory [53, 54].”
The intriguing role of module criticality in the generalization of deep networks (2019, methodology): “…(Neyshabur et al., 2015; Bartlett et al., 2017; Neyshabur et al., 2019; Long & Sedghi, 2019; Wei & Ma, 2019), or they use approaches similar to PAC-Bayes (McAllester, 1999; Dziugaite & Roy, 2017; Neyshabur et al., 2017; 2018; Arora et al., 2018; Nagarajan & Kolter, 2019a; Zhou et al., 2019).”
PAC-Bayesian Analysis of Co-clustering and Beyond (2010, methodology): “Following the pioneering work of McAllester (1998, 1999), the PAC-Bayesian bounds were tightened and simplified by Seeger (2002, 2003).”
Chaining Mutual Information and Tightening Generalization Bounds (2018, background): “Generalization bounds have evolved throughout the years, starting from the basic union bound over the hypothesis set, the refined union bound, VC-dimension and Rademacher complexity [2]; and algorithm dependent bounds such as PAC-Bayesian bounds [3], uniform”
On the Complexity of Linear Prediction: Risk Bounds, Margin Bounds, and Regularization (2008, background): “We now show that (a form of) the PAC Bayesian theorem [McAllester, 1999] is a consequence of Theorem 3.”