VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY

By G. Brier. 1950

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, background): “(4) (the multiclass Brier score [5]) because, unlike the cross-entropy, it is bounded and less sensitive to incorrect predictions.”
Calibrate Before Use: Improving Few-Shot Performance of Language Models (2021, background): “This goal is distinct from statistical calibration (Brier, 1950; Guo et al., 2017), i.e., aligning a model’s conﬁdence estimate with its true accuracy.”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, methodology): “The quality of calibration can be measured by proper scoring rules (Gneiting and Raftery, 2007) such as log predictive probabilities and the Brier score (Brier, 1950).”
Verified Uncertainty Calibration (2019, methodology): “The objective is to minimize the mean-squared error, also known as the Brier score [22], subject to a calibration error budget [5].”
Measuring Calibration in Deep Learning (2019, methodology): “Unlike the Brier score (Brier, 1950) and Hosmer-Lemeshow (Hosmer & Lemesbow, 1980), we’d like the metric to be scalar-valued in order to easily benchmark methods, and to only measure calibration.”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, background): “Brier Score # (Brier, 1950) Proper scoring rule for measuring the accuracy of predicted probabilities.”
Posterior calibration and exploratory analysis for natural language processing models (2015, background): “…data, such as the familiar cross entropy (negative average log-likelihood), or mean squared error, also known as the Brier score when y is binary (Brier, 1950), Both tend to attain better (lower) values when q is near 1 when y = 1 , and near 0 when y = 0 ; and they achieve a perfect value of 0…”
ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring (2019, methodology): “A standard cross-entropy loss is used for labeled data, whereas the loss for unlabeled data is computed using a mean square error (i.e. the Brier score (Brier, 1950)) and is weighted with a hyperparameter λ U .”
Deep Ensembles: A Loss Landscape Perspective (2019, methodology): “[2019], we measure accuracy as well as Brier score [Brier, 1950] (lower values indicate better uncertainty estimates).”
Efficient Sequential Decision Making (2017, background): “The former is traditionally popular in meteorology [10] and the latter has connections with online Gaussian density estimation [50].”
Stated Beliefs and Play in Normal Form Games (2008, unknown): “Brier, G. W. (1950): Verification of Forecasts Expressed in Terms of Probability, Monthly Weather Review, 78, 1-3. Camerer, Colin, Ho, Teck, and Chong, Juin-Kuan (2004), A Cognitive Hierarchy Model of Games, forthcoming, Quarterly Journal of Economics. Camerer, Colin, Ho, Teck, Chong, Juin-Kuan,”
PROPER LOCAL SCORING RULES ON DISCRETE SAMPLE SPACES (2011, unknown): “Examples of proper scoring rules are the Brier score S(x, p) = ‖p‖(2)−2px [Brier (1950)], where ‖p‖ 2 = ∑ x p 2 x, with 1-homogeneous entropy functionH(α) =−‖α‖(2)/α+ and the spherical score S(x,p) =−px/‖p‖ with 1-homogeneous entropy function H(α) = −‖α‖ [Good (1971), Dawid (2007)].”
A Hybrid Ensemble Kalman Filter-3D Variational Analysis Scheme (2000, methodology): “The quality of the forecasts were evaluated using Brier scores (Brier 1950; W ilks 1995), a measure of the mean-squared error of probabilistic forecasts.”
Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration (2019, background): “Meteorologists were among the first to think about calibration, with [3] introducing an evaluation measure for probabilistic forecasts, which we now call Brier score; [21] proposing reliability diagrams, which allow us”
Predicting mortgage default using convolutional neural networks (2018, methodology): “In addition to Accuray, Sensitivity, Specificity and AUC, the Brier Score (Brier, 1950) and the H-measure (Hand, 2009, 2010) are reported.”