Suboptimal behavior of Bayes and MDL in classification under misspecification

By Peter D. Grünwald, J. Langford. 2004

82 citations recorded by the source corpus.

We show that forms of Bayesian and MDL inference that are often applied to classification problems can be inconsistent. This means that there exists a learning problem such that for all amounts of data the generalization errors of the MDL classifier and the Bayes classifier relative to the Bayesian posterior both remain bounded away from the smallest achievable generalization error. From a Bayesian point of view, the result can be reinterpreted as saying that Bayesian inference can be inconsistent under misspecification, even for countably infinite models. We extensively discuss the result from both a Bayesian and an MDL perspective.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods, Statistical Learning Theory, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

In Statistical Learning Theory

In Signal Processing

Citations in the corpus

PAC-Bayesian Theory Meets Bayesian Inference (2016, neutral): “See also Grünwald and Langford [17], Lacoste-Julien et al. [26], Meir and Zhang [35], Ng and Jordan [36], Rousseau [40] for other studies drawing links between frequentist statistics and Bayesian inference, but outside the PAC-Bayesian framework.”
Follow the leader if you can, hedge if you must (2013, methodology): “83 using techniques by Györfi and Ottucsák (2007) that are essentially equivalent to our Lemma 2 below; Gerchinovitz (2011, Remark 2.”
Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It (2014, methodology): “The setup is inspired by the work of [12, 13], who gave a mathematical proof that Bayesian inference can be inconsistent under misspecification in a related but much more artificial classification setting.”
Philosophy and the practice of Bayesian statistics (2010, unknown): “They may not even be the parameter values which give the best predictions (Grünwald and Langford, 2007; Müller, 2011).”
The Safe Bayesian - Learning the Learning Rate via the Mixability Gap (2012, methodology): “Yet when the model is incorrect, in the setting of Grünwald and Langford (2007), unlike standard Bayes, the safe Bayesian posterior does learn to predict optimally, i.e. as well as the single distribution in the model that predicts best.”