Model averaging in ecology: a review of Bayesian, information-theoretic, and tactical approaches for predictive inference

By Carsten F. Dormann, Justin M. Calabrese, G. Guillera‐Arroita, E. Matechou, V. Bahn, Kamil A. Barton, Colin M. Beale, Simone Ciuti, Jane Elith, Katharina Gerstner and 13 more. Ecological Monographs, 2018

342 citations recorded by the source corpus.

In ecology, the true causal structure for a given problem is often not known, and several plausible models and thus model predictions exist. It has been claimed that using weighted averages of these models can reduce prediction error, as well as better reflect model selection uncertainty. These claims, however, are often demonstrated by isolated examples. Analysts must better understand under which conditions model averaging can improve predictions and their uncertainty estimates. Moreover, a large range of different model averaging methods exists, raising the question of how they differ in their behaviour and performance. Here, we review the mathematical foundations of model averaging along with the diversity of approaches available. We explain that the error in model‐averaged predictions depends on each model's predictive bias and variance, as well as the covariance in predictions between models, and uncertainty about model weights. We show that model averaging is particularly useful if the predictive error of contributing model predictions is dominated by variance, and if the covariance between models is low. For noisy data, which predominate in ecology, these conditions will often be met. Many different methods to derive averaging weights exist, from Bayesian over information‐theoretical to cross‐validation optimized and resampling approaches. A general recommendation is difficult, because the performance of methods is often context dependent. Importantly, estimating weights creates some additional uncertainty. As a result, estimated model weights may not always outperform arbitrary fixed weights, such as equal weights for all models. When averaging a set of models with many inadequate models, however, estimating model weights will typically be superior to equal weights. We also investigate the quality of the confidence intervals calculated for model‐averaged predictions, showing that they differ greatly in behaviour and seldom manage to achieve nominal coverage. Our overall recommendations stress the importance of non‐parametric methods such as cross‐validation for a reliable uncertainty quantification of model‐averaged predictions.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/47b44fcc3493f29989caa68139d4f477ebd1a965

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

A standard protocol for reporting species distribution models (2020, methodology): “Model averaging refers to situations where different models are fit and then combined into a single prediction, which could be desirable when several candidate models are similarly plausible or because several alternative modelling approaches are available (Hastie et al. 2009, Dormann et al. 2018).”
Climate change vulnerability assessment of species (2018, background): “…to as the model ‘ ensemble ’ , may be averaged to produce a single projection, with the degree of agreement between projections represented by a measure of ‘ spread ’ such as the standard deviation or coefficient of variation (for details and caveats of model averaging, see Dormann et al. (2018)).”
Model complexity affects species distribution projections under climate change (2019, background): “Projection ensembles are particularly useful to quantify uncertainty and to obtain consensus projections, which are arguably superior to single model projections (Araújo & New, 2007, but see Dormann et al., 2018).”
Hierarchical generalized additive models in ecology: an introduction with mgcv (2019, background): “Predictive accuracy may also be substantially improved by averaging over multiple models ( Dormann et al., 2018 ).”
A review of evidence about use and performance of species distribution modelling ensembles like BIOMOD (2019, methodology): “Ensembles combine models with the intention of obtaining bet‐ ter separation of the signal from noise (Araújo & New, 2007; Dormann et al., 2018).”
Model Averaging and Its Use in Economics (2017, background): “ reviews and discussions; for example by Hoeting et al. (1999) on BMA, Clyde and George (2004) on model uncertainty, Moral-Benito (2015) on model averaging in economics and Wang et al. (2009) on FMA. Dormann et al. (2018) present an elaborate survey of model averaging methods used in ecology. A rece”