A general framework for updating belief distributions

By Pier Giovanni Bissiri, Christopher C. Holmes, Stephen G. Walker. 2013

633 citations recorded by the source corpus.

We propose a framework for general Bayesian inference. We argue that a valid update of a prior belief distribution to a posterior can be made for parameters which are connected to observations through a loss function rather than the traditional likelihood function, which is recovered as a special case. Modern application areas make it increasingly challenging for Bayesians to attempt to model the true data‐generating mechanism. For instance, when the object of interest is low dimensional, such as a mean or median, it is cumbersome to have to achieve this via a complete model for the whole data distribution. More importantly, there are settings where the parameter of interest does not directly index a family of density functions and thus the Bayesian approach to learning about such parameters is currently regarded as problematic. Our framework uses loss functions to connect information in the data to functionals of interest. The updating of beliefs then follows from a decision theoretic approach involving cumulative loss functions. Importantly, the procedure coincides with Bayesian updating when a true likelihood is known yet provides coherent subjective inference in much more general settings. Connections to other inference frameworks are highlighted.

Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

PAC-Bayesian Theory Meets Bayesian Inference (2016, neutral): “The main assumption of frequentist learning theories—including PAC-Bayes—is that (X,Y ) is (1)Some existing connections [3, 7, 16, 25, 42, 43, 49] are discussed in Appendix A.”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, methodology): “Tempering posteriors is a well-known practice in statistics, where it goes by the names Safe Bayes, generalized Bayesian inference, and fractional Bayesian inference (e.g., de Heide et al., 2019; Grünwald et al., 2017; Barron & Cover, 1991; Walker & Hjort, 2001; Zhang, 2006; Bissiri et al., 2016; Gr”
Communication-Efficient Distributed Statistical Inference (2016, methodology): “Extensions to the Gibbs posterior [Bissiri et al., 2013] where L is replaced with a generic loss function L in posterior (15) is straightforward.”
General Bayesian updating and the loss-likelihood bootstrap (2017, methodology): “In Bissiri et al. (2016) a number of possible alternative means of calibration are discussed, including subjective calibration, unit information loss matching and hierarchical treatment via a loss function onw. Although these ideas may suit particular applications, they are less well motivated in ge”
Bayesian Probabilistic Numerical Methods (2017, methodology): “…belief distribution µ in response to the information A(x) = a obtained about the unknown x. Generic approaches to update belief distributions, which generalise Bayesian inference beyond the unique update demanded in Bayes theorem, were formalised in Bissiri et al. (2016); de Carvalho et al. (2017).”
Calibrating general posterior credible regions (2015, methodology): “. Similar questions about scaling posterior distributions to address model misspeci- cation have been considered recently in the literature. Ideas for choosing the posterior scaling are presented in Bissiri et al. (2016) and Holmes and Walker (2017), including hierarchical Bayes and loss/information”
Communication-efficient distributed statistical learning (2016, methodology): “Extensions to the Gibbs posterior Bissiri Pier and Stephen [2013] where L is replaced with a generic loss function L in posterior (14) is also straightforward.”
Bayesian forecasting in economics and finance: A modern review (2022, methodology): “Adopting the exponential updating rule proposed by Bissiri et al. (2016) (see also Giummolè et al., 2017, Holmes and Walker, 2017, Guedj, 2019, Lyddon et al., 2019, and Syring and Martin, 2019), Loaiza-Maya et al. (2021) define the generalized (or Gibbs) posterior: πw(θ|y1:T ) = exp [wST (θ)] π(θ)”
Robust generalised Bayesian inference for intractable likelihoods (2021, positive): “A particularly versatile approach to robustness, which encompasses most of the above, is generalised Bayesian inference (Bissiri et al., 2016) (see also the earlier work of Chernozhukov & Hong, 2003).”
On the properties of variational approximations of Gibbs posteriors (2015, background): “…places, and under different motivations: in Econometrics, as a way to avoid direct maximisation in moment estimation [Chernozhukov and Hong, 2003]; and in Bayesian decision theory, as as way to define a Bayesian posterior distribution when no likelihood has been specified [Bissiri et al., 2013].”
Bayesian fractional posteriors (2016, methodology): “The fractional posteriors are a special instance of Gibbs posteriors [31] or quasi-posteriors [15], where the negative exponent of a loss function targeted towards a specific parameter of interest is used as a surrogate for the likelihood function; see [10] for a general framework for updating of pr”
Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It (2014, background): “…posterior is a notion going back at least to Vovk (1990) and has been developed mainly within the so-called (frequentist) PAC-Bayesian framework McAllester (2003), Seeger (2002), Catoni (2007), Audibert (2004), Zhang (2006b); see also Bissiri et al. (2016) and the discussion in Section 7.”
User-friendly introduction to PAC-Bayes bounds (2021, background): “Gibbs posteriors are discussed in a decision theoretic framework by [34].”
Approximate Models and Robust Decisions (2014, background): “Note that it may be sensible to match the prior to the decision one hopes to reach; the prior should represent “an adversary who will need to be disillusioned by the data to stop further experimentation” Spiegelhalter, Freedman and Parmar (1994), as quoted in Berry et al.”
Martingale posterior distributions (2021, methodology): “A more general framework is the decision task discussed in Bissiri et al. (2016), where the aim is to minimize a functional of an unknown distribution function F0 from which samples Y1:n are i.i.d..”