Making and Evaluating Point Forecasts

By Tilmann Gneiting. 2009

1,180 citations recorded by the source corpus.

Typically, point forecasting methods are compared and assessed by means of an error measure or scoring function, with the absolute error and the squared error being key examples. The individual scores are averaged over forecast cases, to result in a summary measure of the predictive performance, such as the mean absolute error or the mean squared error. I demonstrate that this common practice can lead to grossly misguided inferences, unless the scoring function and the forecasting task are carefully matched. Effective point forecasting requires that the scoring function be specified ex ante, or that the forecaster receives a directive in the form of a statistical functional, such as the mean or a quantile of the predictive distribution. If the scoring function is specified ex ante, the forecaster can issue the optimal point forecast, namely, the Bayes rule. If the forecaster receives a directive in the form of a functional, it is critical that the scoring function be consistent for it, in the sense that the expected score is minimized when following the directive. A functional is elicitable if there exists a scoring function that is strictly consistent for it. Expectations, ratios of expectations and quantiles are elicitable. For example, a scoring function is consistent for the mean functional if and only if it is a Bregman function. It is consistent for a quantile if and only if it is generalized piecewise linear. Similar characterizations apply to ratios of expectations and to expectiles. Weighted scoring functions are consistent for functionals that adapt to the weighting in peculiar ways. Not all functionals are elicitable; for instance, conditional value-at-risk is not, despite its popularity in quantitative finance.

Subfields: Applied Mathematics, Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/06f682919eafa8adbcc1b7bec23516cb962e7b22

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Mathematics

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Real-Time Inflation Forecasting in a Changing World (2009, positive): “Gneiting (2011) showed that the RMSE is only a consistent evaluation measure when the point forecast equals the mean of the predictive distribution, and we therefore base the RMSE of a model on the mean of the corresponding distribution of inflation predictions.”
A guide to Bayesian model selection for ecologists (2015, methodology): “Out-of-sample validation relies on the ability to compute a similarity statistic or scoring rule to obtain a measure of closeness between our out-of-sample data yoos and the predictions ŷoos (e.g., Bernardo 1979, Czado et al. 2009, Gneiting and Raftery 2007, Gneiting 2011).”
Long-term Forecasting with TiDE: Time-series Dense Encoder (2023, background): “Note that different predictive statistics are optimal for different target metrics (Awasthi et al., 2021; Gneiting, 2011) and therefore we should look at a target metric that is closely aligned with the training loss in this case.”
Spatial Postprocessing of Ensemble Forecasts for Temperature Using Nonhomogeneous Gaussian Regression (2014, methodology): “semble model output statistics method for surface temperature, originally proposed by Gneiting et al. (2005). NGR is a parsimonious postprocessing technique that, for temperature, returns a Gaussian predictive distri-”
Frontiers in VaR forecasting and backtesting (2016, background): “Gneiting (2011) points out that the loss functions proposed by Lopez (1999) and extended by Caporin (2008) are not of the GPL form and, consequently, this could explain why these functions are not able to properly distinguish between the true Data Generating Process (DGP) and alternative models to…”
On the Comparison of Interval Forecasts (2018, background): “It can be shown Gneiting and Raftery (2007) that the optimal interval forecast under Winkler loss has left and right endpoints given by the 1∕λl and 1−1∕λu quantiles, so that for the evaluation of intervals with (1−α) coverage, one should set 1∕λl + 1∕λu = α. A popular choice for λl and λu in practi”
Higher order elicitability and Osband’s principle (2015, methodology): “Finally, ESα fails to be 1-elicitable [Gneiting (2011), Weber (2006)], whereas VaRα is 1-elicitable for most classes of distributions F of practical relevance.”
Expected Shortfall is jointly elicitable with Value at Risk - Implications for backtesting (2015, background): “e continues to be lively debate about the appropriate choice of a quantitative risk measure for regulatory purposes or internal risk management. In this context, it has been shown by Weber (2006) and Gneiting (2011) that Expected Shortfall (ES) is not elicitable. Speciﬁcally, there is no strictly co”
Variogram-Based Proper Scoring Rules for Probabilistic Forecasts of Multivariate Quantities* (2015, background): “It has been demonstrated that the use of improper scores can lead to misguided inferences about predictive performance (Gneiting 2011).”
A better measure of relative prediction accuracy for model selection and model estimation (2015, neutral): “According to a number of surveys reviewed by Gneiting (2011), MAPE is the most commonly”
COHERENCE AND ELICITABILITY (2013, background): “Gneiting (2011) has shown that ES is not elicitable, which may be a partial explanation for the difficulties with robust estimation and backtesting.”
Ensemble Model Output Statistics for Wind Vectors (2012, methodology): “The absolute error is defined as ae(P, y) 5 jmedP 2 yj, (A3) where medP is a median of the probability distribution P (Gneiting 2011; Pinson and Hagedorn 2012).”
An Academic Response to Basel 3.5 (2014, background): “The relevant notion here is elicitability ; see Gneiting (2011).”
Expectile Asymptotics (2015, background): “(6) Here, any other scoring function for the expectile (Gneiting, 2011) could be used instead, they all result in the same estimator, the expectile of the empirical distribution function.”