Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It

By Peter D. Grünwald, Thijs van Ommen. 2014

312 citations recorded by the source corpus.

We empirically show that Bayesian inference can be inconsistent under misspecification in simple linear regression problems, both in a model averaging/selection and in a Bayesian ridge regression setting. We use the standard linear model, which assumes homoskedasticity, whereas the data are heteroskedastic, and observe that the posterior puts its mass on ever more high-dimensional models as the sample size increases. To remedy the problem, we equip the likelihood in Bayes' theorem with an exponent called the learning rate, and we propose the Safe Bayesian method to learn the learning rate from the data. SafeBayes tends to select small learning rates as soon the standard posterior is not `cumulatively concentrated', and its results on our data are quite encouraging.

Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, methodology): “Guedj, 2019), with distributions over the parameters. Langford & Caruana (2002) devised a PAC-Bayes generalization bound for small stochastic neural networks (two layer with two hidden units) achieving an improvement over the existing deterministic generalization bounds. Dziugaite & Roy (2017) exten”
Robust generalised Bayesian inference for intractable likelihoods (2021, unknown): “…limited to) strategies for the robust specification of prior belief (Berger et al., 1994), the so-called safe Bayes approach (Grünwald, 2011, 2012; Grünwald & van Ommen, 2017; de Heide et al., 2020), power posteriors (e.g. Holmes & Walker, 2017), coarsened posteriors (Miller & Dunson, 2019),…”
Bayesian fractional posteriors (2016, background): “In fact, [26] empirically demonstrate through a detailed simulation study that even convergence to the nearest KL point may not take place in misspecified models.”
User-friendly introduction to PAC-Bayes bounds (2021, background): “, Xn) is not consistent, a tempered posterior for α small enough will be [86].”