PAC-Bayesian Theory Meets Bayesian Inference

By Pascal Germain, Francis R. Bach, Alexandre Lacoste, Simon Lacoste-Julien. 2016

214 citations recorded by the source corpus.

We exhibit a strong link between frequentist PAC-Bayesian risk bounds and the Bayesian marginal likelihood. That is, for the negative log-likelihood loss function, we show that the minimization of PAC-Bayesian generalization risk bounds maximizes the Bayesian marginal likelihood. This provides an alternative explanation to the Bayesian Occam's razor criteria, under the assumption that the data is generated by an i.i.d distribution. Moreover, as the negative log-likelihood is an unbounded loss function, we motivate and propose a PAC-Bayesian theorem tailored for the sub-gamma loss family, and we show that our approach is sound on classical Bayesian linear regression tasks.

Subfields: Artificial Intelligence, Computer Networks and Communications, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Probabilistic & Bayesian Methods

Citations in the corpus

Predictive Multiplicity in Classification (2019, methodology): “…explicitly represent uncertainty over the set of competing models by means of a posterior probability distribution, but then use the posterior distribution to construct a single model for deployment [e.g., via a majority vote or a randomization procedure as in PAC Bayesian approaches, 18, 32].”
A Bayesian Perspective on Generalization and Stochastic Gradient Descent (2017, background): “Germain et al. (2016) showed that maximizing the evidence (or “marginal likelihood”) minimizes a PAC-Bayes generalization bound.”
Generalizing from a Few Examples (2019, background): “For this smaller H̃ , Dtrain is sufficient to learn a reliable hI [43, 86, 99].”
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy (2017, background): “Germain et al. (2016) described PAC-Bayes generalization bounds for unbounded loss functions.”
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priors (2017, background): “Germain et al. (2016) described PAC-Bayes generalization bounds for unbounded loss functions, though it requires that one make additional assumptions about the distribution of the empirical risk, which we would prefer not to make.”
Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory (2017, background): “Using (22) and (19), the optimization problem can be reformulated in an equivalent form 16 The log-likelihood is analogous to the loss function PAC-Bayesian analysis (Germain et al., 2016).”
Appendix for “Learning Overlapping Representations for the Estimation of Individualized Treatment Effects” (2020, methodology): “The expected factual risk can be bounded using some existing PAC-Bayes theorems in supervised learning [2, 6].”
Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints (2017, methodology): “Germain et al. (2016) extended previous results to sub-Gaussian losses, but their result introduced an extra additive error term 1 2 s 2 , where s 2 is the Sub-Gaussian variance factor.”
Deep learning generalizes because the parameter-function map is biased towards simple functions (2018, background): “The main quantity in the PAC-Bayes theorem, P ( U ) , is precisely the probability of a given set of labels for the set of instances in the training set, also known as marginal likelihood , a connection explored in recent work (Smith & Le (2018); Germain et al. (2016)).”
User-friendly introduction to PAC-Bayes bounds (2021, background): “This connection is for example discussed in [15, 79].”