Strictly Proper Scoring Rules, Prediction, and Estimation

By Tilmann Gneiting, Adrian E. Raftery. 2007

6,677 citations recorded by the source corpus.

Subfields: Atmospheric Science, Global and Planetary Change, Management Science and Operations Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atmospheric Science

In Global and Planetary Change

In Management Science and Operations Research

In Evaluation & Benchmarks

Citations in the corpus

f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization (2016, methodology): “The advantage of VAEs is in the encoder model which allows efficient inference from observation to latent representation and overall they are a compelling alternative to f -GANs and recent work has studied combinations of the two approaches [23] As an alternative to the GAN training objective the wo”
Demystifying MMD GANs (2018, methodology): “An insight comes from considering the score function associated with the energy distance, which we now brieﬂy review (Gneiting & Raftery, 2007).”
Classification Accuracy Score for Conditional Generative Models (2019, background): “Second, since the losses considered here are not proper scoring rules [38], one could obtain reasonable CAS from suboptimal inference networks.”
How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary? (2015, background): “In statistics, this property is referred to as strictly proper scoring rule estimation (Gneiting & Raftery, 2007).”
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, background): “To this end, a theoretically more well-founded uncertainty metric is to examine strictly proper scoring rules [25] s(.”
High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes (2019, background): “(a) Effect of the number of evaluation samples ( num eval samples ) on the CRPS for electricity, solar, and taxi datasets.”
A Spectral Energy Distance for Parallel Speech Synthesis (2020, positive): “As we show there, the analysis of Gneiting and Raftery [10, Theorem 5.1] can be used to show that this choice makes (8) a strictly proper scoring rule for learning q θ ( x | c ) with respect to the ground-truth conditional distribution over spectrograms, meaning that L GED ( q ) > L GED ( p ) for an”
Measuring Calibration in Deep Learning (2019, background): “For example, Brier score measures the mean squared difference between the predicted probability and the actual outcome (Gneiting and Raftery, 2007).”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, background): “Each proper scoring rule is also associated with a corresponding entropy function and Shannon entropy is that for log probability (Gneiting & Raftery, 2007).”
Posterior calibration and exploratory analysis for natural language processing models (2015, background): “These two loss functions are instances of proper scoring rules (Gneiting and Raftery, 2007; Bröcker, 2009).”
Accurate Uncertainties for Deep Learning Using Calibrated Regression (2018, background): “The study of calibration originates in the statistics literature (Murphy, 1973; Dawid, 1984), mainly in the context of evaluating probabilistic forecasts using proper loss functions (Gneiting and Raftery, 2007).”
The Cramer Distance as a Solution to Biased Wasserstein Gradients (2017, methodology): “As a replacement we propose the Cramér distance (Székely, 2002; Rizzo and Székely, 2016), also known as the continuous ranked probability score in the probabilistic forecasting literature (Gneiting and Raftery, 2007).”
Training Agents using Upside-Down Reinforcement Learning (2019, methodology): “Using a loss function L , it can be estimated by solving the following SL problem: where Here len( τ ) is the length of any trajectory τ , and in general − L can be any proper score function [Gneiting and Raftery, 2007].”