Exascale Deep Learning for Climate Analytics

By Thorsten Kurth, Sean Treichler, J. Romero, Mayur Mudigonda, Nathan Luehr, E. Phillips, A. Mahesh, Michael A. Matheson, Jack R. Deslippe, M. Fatica and 2 more. 2018

284 citations recorded by the source corpus.

We extract pixel-level masks of extreme weather patterns using variants of Tiramisu and DeepLabv3+ neural networks. We describe improvements to the software frameworks, input pipeline, and the network training algorithms necessary to efficiently scale deep learning on the Piz Daint and Summit systems. The Tiramisu network scales to 5300 P100 GPUs with a sustained throughput of 21.0 PF/s and parallel efficiency of 79.0%. DeepLabv3+ scales up to 27360 V100 GPUs with a sustained throughput of 325.8 PF/s and a parallel efficiency of 90.7% in single precision. By taking advantage of the FP16 Tensor Cores, a half-precision version of the DeepLabv3+ network achieves a peak and sustained throughput of 1.13 EF/s and 999.0 PF/s respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

PPINN: Parareal Physics-Informed Neural Network for time-dependent PDEs (2019, background): “In the past decade, the large computational cost for training DNN has been mitigated by a number of advances, including high-performance computers [3], graphics processing units (GPUs) [4], tensor processing units (TPUs) [5], and fast large-scale optimization schemes [6], i.e., adaptive moment…”
Tackling Climate Change with Machine Learning (2019, background): “ because extreme events are, by deﬁnition, rare. ML has been used successfully to classify some extreme weather events. Researchers have used deep learning to classify [501], detect [480] and segment [502] cyclones and atmospheric rivers, as well as tornadoes [503], in historical climate datasets. T”
WeatherBench: A Benchmark Data Set for Data‐Driven Weather Forecasting (2020, background): “How to produce probabilistic data-driven”
Heterogeneity-Aware Asynchronous Decentralized Training (2019, methodology): “More recently, All-Reduce [41], an alternative distributed solution utilizing the advanced Ring All-Reduce algorithm [15], is shown to provide superior performance than PS [26, 31, 45, 52].”
Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUs (2019, background): “[4], [5].”
Can Machines Learn to Predict Weather? Using Deep Learning to Predict Gridded 500‐hPa Geopotential Height From Historical Weather Data (2019, background): “Deep NNs have also been used to identify extreme weather and climate patterns in observed and modeled atmospheric states (Kurth et al., 2018; Lagerquist, McGovern, & Gagne, 2019; Liu et al., 2016), improve parameteriza-46 tions in GCMs (e.g., Brenowitz & Bretherton, 2018; Rasp, Pritchard, &…”
Improving Data‐Driven Global Weather Prediction Using Deep Convolutional Neural Networks on a Cubed Sphere (2020, background): “Deep NNs have also been used to identify extreme weather and climate patterns in observed and modeled atmospheric states [24, 19, 20], to predict extreme weather events [e.”
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames (2019, background): “As a comparison, Savva et al. (2019) reached 75 million steps (an order of magnitude more than prior work) in 2.5 days using 2 GPUs – at that rate, it would take them over a month (wall-clock time) to achieve the scale of our study.”
Etalumis: Bringing Probabilistic Programming to Scientific Simulators at Scale (2019, unknown): “mber draws and conditioning, respectively. enable rapid prototyping, model evaluation, and making use of HPC capacity, scaling deep learning training to multiple computation units is highly desirable [38, 44, 45, 51, 52]. In this context there are three prominent parallelism strategies: data- and mo”
Deep Learning for Spatio-Temporal Data Mining: A Survey (2019, background): “to incorporate deep learning models for analyzing various weather and environment data [79], [129], such as air quality inference [19], [94], precipitation prediction [100], [161], wind speed prediction [96], [200], and extreme weather detection [100].”
Visualizing the Consequences of Climate Change Using Cycle-Consistent Adversarial Networks (2019, background): “Finally, climate science-motivated discovery could lead to advances in ML, as demonstrated in the application of deep learning methods for pixel-level segmentation of extreme events by Kurth et al. (2018).”
Graph Neural Controlled Differential Equations for Traffic Forecasting (2021, background): “…applications, ranging from trafﬁc to climate forecasting (Zaytar and El Amrani 2016; Shi et al. 2015, 2017; Liu et al. 2016; Racah et al. 2016; Kurth et al. 2018; Cheng et al. 2018a,b; Hossain et al. 2015; Ren et al. 2021; Tekin et al. 2021; Li et al. 2018; Yu, Yin, and Zhu 2018; Wu et al.…”
Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics (2019, background): “…in fields ranging from weather and climate prediction (Gneiting & Raftery, 2005; Weyn, Durran, & Caruana, 2019), medicine analytics (Esteva et al., 2017; Kurth et al., 2018), to modeling ocean dynamics and turbulent flows (Aksamit, Sapsis, & Haller, 2019; Brunton, Noack, & Koumoutsakos, 2020).”