On the properties of variational approximations of Gibbs posteriors

By Pierre Alquier, James Ridgway, Nicolas Chopin. 2015

286 citations recorded by the source corpus.

The PAC-Bayesian approach is a powerful set of techniques to derive non- asymptotic risk bounds for random estimators. The corresponding optimal distribution of estimators, usually called the Gibbs posterior, is unfortunately intractable. One may sample from it using Markov chain Monte Carlo, but this is often too slow for big datasets. We consider instead variational approximations of the Gibbs posterior, which are fast to compute. We undertake a general study of the properties of such approximations. Our main finding is that such a variational approximation has often the same rate of convergence as the original PAC-Bayesian procedure it approximates. We specialise our results to several learning tasks (classification, ranking, matrix completion),discuss how to implement a variational approximation in each case, and illustrate the good properties of said approximation on real datasets.

Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

PAC-Bayesian Theory Meets Bayesian Inference (2016, methodology): “Indeed, Equation (18) becomes E θ∼ρ̂∗ L`nllD (θ) = E η∼ρ̂∗0 E θ∼ρ̂∗η L`nllD (θ) ≤ 1 2(1−c) s 2 − 1n ln ( E η∼π0 ZX,Y,η δ ) .”
PAC-Bayesian Bounds based on the Rényi Divergence (2016, result): “We can also recover bounds that are similar to the ones of Pentina and Lampert [2015] and Alquier et al. [2015] by considering a linear function ∆lin(q, p) = p − q. In the transductive learning setting [Vapnik, 1998], where one has access to a subset of m labeled examples drawn from a set of N examp”
Calibrating general posterior credible regions (2015, methodology): “(2016), Alquier et al. (2016), Zhang (2006), Jiang and Tanner (2008), and Syring and Martin (2017). In such cases, a scale—or inverse temperature—parameter must be specified to properly weight the information in the data relative to that in the prior, but this ultimately boils down to tuning the Gib”
Appendix for “Learning Overlapping Representations for the Estimation of Individualized Treatment Effects” (2020, methodology): “(Alquier et al. [2]) Given a distribution P over X × Y , a hypothesis space F , a loss function l : F × X × Y → R , a prior distribution π over F , a δ ∈ (0 , 1] , and a real number κ > 0 , with probability at least 1 − δ over the random choice S = ( X , Y ) ∈ P n , ∀ ˆ ρ on F , we have where Note…”
User-friendly introduction to PAC-Bayes bounds (2021, methodology): “Since [82], we used in our paper with James Ridgway and Nicolas Chopin [9] a similar approach: we defined a “Hoeffding assumption” and a “Bernstein assumtpion” that corresponds to data satisfying a Hoeffding type inequality, or a Bernstein type inequality + the usual Bernstein condition (Definition ”