Statistical Methods for Eliciting Probability Distributions

By Paul H. Garthwaite, Joseph B. Kadane, Anthony OʼHagan. 2005

869 citations recorded by the source corpus.

Elicitation is a key task for subjectivist Bayesians. Although skeptics hold that elicitation cannot (or perhaps should not) be done, in practice it brings statisticians closer to their clients and subject-matter expert colleagues. This article reviews the state of the art, reflecting the experience of statisticians informed by the fruits of a long line of psychological research into how people represent uncertain information cognitively and how they respond to questions about that information. In a discussion of the elicitation process, the first issue to address is what it means for an elicitation to be successful; that is, what criteria should be used. Our answer is that a successful elicitation faithfully represents the opinion of the person being elicited. It is not necessarily “true” in some objectivistic sense, and cannot be judged in that way. We see that elicitation as simply part of the process of statistical modeling. Indeed, in a hierarchical model at which point the likelihood ends and the prior begins is ambiguous. Thus the same kinds of judgment that inform statistical modeling in general also inform elicitation of prior distributions. The psychological literature suggests that people are prone to certain heuristics and biases in how they respond to situations involving uncertainty. As a result, some of the ways of asking questions about uncertain quantities are preferable to others, and appear to be more reliable. However, data are lacking on exactly how well the various methods work, because it is unclear, other than by asking using an elicitation method, just what the person believes. Consequently, one is reduced to indirect means of assessing elicitation methods. The tool chest of methods is growing. Historically, the first methods involved choosing hyperparameters using conjugate prior families, at a time when these were the only families for which posterior distributions could be computed. Modern computational methods, such as Markov chain Monte Carlo, have freed elicitation from this constraint. As a result, now both parametric and nonparametric methods are available for low-dimensional problems. High-dimensional problems are probably best thought of as lacking another hierarchical level, which has the effect of reducing the as-yet-unelicited parameter space. Special considerations apply to the elicitation of group opinions. Informal methods, such as Delphi, encourage the participants to discuss the issue in the hope of reaching consensus. Formal methods, such as weighted averages or logarithmic opinion pools, each have mathematical characteristics that are uncomfortable. Finally, there is the question of what a group opinion even means, because it is not necessarily the opinion of any participant.

Subfields: General Decision Sciences, Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/a54fa610535b1cec2a0582e699d31e281ad8eb4f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In General Decision Sciences

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Efficiently encoding and modeling subjective probability distributions for quantitative variables. (2016, background): “It is well established that multiple regression models of human judgments perform better than the humans themselves (Dawes, Faust, & Meehl, 1989; Goldberg, 1970; Hoffman, 1960). In all these studies, the probability judgments were based on multidimensional cues (e.g., Minnesota Multiphasic Personali”
The role of expert opinion in environmental modelling (2012, background): “The literature on expert elicitation uses the term “expert” pragmatically to describe anybody whose opinion might be of interest; in statistics (e.g. Morris, 1974; Garthwaite et al., 2005), ecology (e.”
Strictly Proper Scoring Rules, Prediction, and Estimation (2007, methodology): “In terms of elicitation, the role of scoring rules is to encourage the assessor to make careful assessments and to be honest (Garthwaite, Kadane, and O’Hagan 2005).”
Conjugate Analysis of the Conway-Maxwell-Poisson Distribution (2006, methodology): “…we have created an application to help the practitioner select the hyperparameters a, b, and c or to aid the statistician in eliciting these values from the expert who understands the context of the data (see Kadane and Wolfson (1997) and Garthwaite et al. (2005) for reviews of elicitation issues).”
Nonparametric Scoring Rules (2015, background): “Much of the literature therefore focuses on eliciting summaries such as mean and variance, median, or quantiles, and the challenging case of correlations in the case of multivariate data (Clemen, Fischer, and Winkler 2000; Garthwaite, Kadane, and O’Hagan 2005).”
Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (2023, background): “extensively studied in various fields including statistics (Garthwaite et al., 2005), psychology (van der Gaag et al.”
Good practices for quantitative bias analysis. (2014, background): “Formulating knowledge or beliefs about unknown parameters into a joint probability distribution is called elicitation of the prior distribution.(73) One formal approach is to ask each expert for an interval within which they think the parameter falls and the odds or percentage they would bet on the ”
Bayesian inference of chemical kinetic models from proposed reactions (2015, background): “Furthermore, as described in Section 3, priors may also be shaped by expert elicitation (Garthwaite et al., 2005) or chosen to reflect relative ignorance about the rate parameter values.”