WeatherBench: A Benchmark Data Set for Data‐Driven Weather Forecasting

By Stephan Rasp, Peter Dueben, Sebastian Scher, Jonathan A. Weyn, Soukayna Mouatadid, N. Thuerey. Journal of Advances in Modeling Earth Systems, 2020

725 citations recorded by the source corpus.

Data‐driven approaches, most prominently deep learning, have become powerful tools for prediction in many domains. A natural question to ask is whether data‐driven methods could also be used to predict global weather patterns days in advance. First studies show promise but the lack of a common data set and evaluation metrics make intercomparison between studies difficult. Here we present a benchmark data set for data‐driven medium‐range weather forecasting (specifically 3–5 days), a topic of high scientific interest for atmospheric and computer scientists alike. We provide data derived from the ERA5 archive that has been processed to facilitate the use in machine learning models. We propose simple and clear evaluation metrics which will enable a direct comparison between different methods. Further, we provide baseline scores from simple linear regression techniques, deep learning models, as well as purely physical forecasting models. The data set is publicly available at https://github.com/pangeo‐data/WeatherBench and the companion code is reproducible with tutorials for getting started. We hope that this data set will accelerate research in data‐driven weather forecasting.

Subfields: Atmospheric Science, Environmental Engineering, Global and Planetary Change

View Paper: https://www.semanticscholar.org/paper/fcbb249016782b000dbc659958c5325f38f58b87

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atmospheric Science

In Environmental Engineering

In Global and Planetary Change

Citations in the corpus

WeatherBench 2: A Benchmark for the Next Generation of Data‐Driven Global Weather Models (2023, background): “Absolute metric values can differ significantly between resolutions but the relative differences between different models are consistent across different evaluation resolution (see Figure S9 in Supporting Information S1 for a comparison of IFS HRES scores at different resolutions and consult the Wea”
FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators (2022, methodology): “…FourCastNet was developed in less than a year and has only a fraction of the number of variables and vertical levels compared to NWP, its accuracy is comparable to the IFS model and better than state-of-the-art DL weather prediction models [Weyn et al., 2021, Rasp et al., 2020] on short timescales.”
Physics-informed machine learning: case studies for weather and climate modelling (2021, background): “A ﬁrst step towards this in medium-range weather forecasting is WeatherBench [129].”
ClimaX: A foundation model for weather and climate (2023, positive): “We use the preprocessed version of ERA5 from WeatherBench (Rasp et al., 2020) for finetuning ClimaX.”
Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere (2023, neutral): “The latitude weighted ACC for a forecast variable v at forecast time-step l is defined following Rasp et al. (2020) as follows: ACC(v, l) = ∑ m,n wLat(m)X̃pred(l) [v,m, n] X̃true(l) [v,m, n]√∑ m,n wLat(m) ( X̃pred(l) [v,m, n] )2∑ m,n wLat(m) ( X̃true(l) [v,m, n] )2 , (32) 7On equiangular grids on…”
FuXi: a cascade machine learning forecasting system for 15-day global weather forecast (2023, methodology): “To facilitate intercomparison between different ML models, the WeatherBench benchmark was introduced to evaluate medium-range weather forecasting (i.e., 3-5 days) [5, 6].”
Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers (2020, background): “Beyond engineering applications and medical simulations, a particularly interesting application of our approach is weather prediction [45], where a simple differentiable solver could be augmented with a learned correction function to recover the costly predictions of operational forecasting systems.”
The operational medium-range deterministic weather forecasting can be extended beyond a 10-day lead time (2023, neutral): “Subsequently, we evaluate this model ’ s forecast performance based on the operational analysis data of the year 2022 that are used for both initial ﬁ elds and prediction validation (More details of the training setting and evaluation metrics are presented in Supplementary Notes 2, 4, and 9).”
Can deep learning beat numerical weather prediction? (2021, neutral): “One notable exception is Weatherbench [142].”
GraphCast: Learning skillful medium-range global weather forecasting (2022, methodology): “We evaluated our model’s performance on all variables and levels in the Scorecard (except the ocean wave variables), as well as most of the WeatherBench (Rasp et al., 2020) variables and levels, giving 69 variables and levels in total.”