Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
By Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, Wang-chun Woo. 2015
9,818 citations recorded by the source corpus.
The goal of precipitation nowcasting is to predict the future rainfall intensity in a local region over a relatively short period of time. Very few previous studies have examined this crucial and challenging weather forecasting problem from the machine learning perspective. In this paper, we formulate precipitation nowcasting as a spatiotemporal sequence forecasting problem in which both the input and the prediction target are spatiotemporal sequences. By extending the fully connected LSTM (FC-LSTM) to have convolutional structures in both the input-to-state and state-to-state transitions, we propose the convolutional LSTM (ConvLSTM) and use it to build an end-to-end trainable model for the precipitation nowcasting problem. Experiments show that our ConvLSTM network captures spatiotemporal correlations better and consistently outperforms FC-LSTM and the state-of-the-art operational ROVER algorithm for precipitation nowcasting.
Subfields: Atmospheric Science, Environmental Engineering, Global and Planetary Change, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Atmospheric Science
- Philip S. Yu
- Richard S. Zemel
- Mengye Ren
- Sungjin Ahn
- Jindong Jiang
- Kevin Trebing
- Siamak Mehrkanoon
- Sergey Levine
In Environmental Engineering
- Raquel Urtasun
- Philip S. Yu
- Bo Li
- Jiaya Jia
- Philipp Krahenbuhl
- Vladlen Koltun
- Sungjin Ahn
- Jindong Jiang
In Global and Planetary Change
- Philip S. Yu
- Sungjin Ahn
- Jindong Jiang
- Vladlen Koltun
- Richard S. Zemel
- Mengye Ren
- Kevin Trebing
- Siamak Mehrkanoon
In Model Architectures
- Yoshua Bengio
- Quoc V. Le
- Ruslan Salakhutdinov
- Kyunghyun Cho
- Alan L. Yuille
- J. Urgen Schmidhuber
- Vladlen Koltun
- Jiashi Feng
Citations in the corpus
- Factorization tricks for LSTM networks (2017, methodology): “In case of LSTM networks, ConvLSTM (Shi et al., 2015), has been introduced to better exploit possible spatiotemporal correlations, which is conceptually similar to grouping.”
- Neural GPUs Learn Algorithms (2015, methodology): “In image processing, convolutional LSTMs , an architecture similar to the Neural GPU, have recently been used for weather prediction (Shi et al., 2015) and image compression (Toderici et al., 2016).”
- Scaling Autoregressive Video Models (2019, background): “Earlier work on video generation mostly focused on deterministic approaches [33, 4, 44, 24, 17], which fail to capture the high degree of stochasticity inherent in video.”
- Recurrent Neural Networks for Semantic Instance Segmentation (2017, methodology): “Romera-Paredes & Torr [17] train a model composed of Convolutional LSTMs [21] that receives convolutional features from a pretrained FCN [12] and outputs the separate object segments for the image.”
- Recurrent Instance Segmentation (2015, background): “A similar recurrent unit has been recently proposed in [36] in the context of weather forecasting in a region.”
- Video Pixel Networks (2016, methodology): “The outputs of the encoders are combined over time with a convolutional LSTM that also preserves the resolution (Hochreiter and Schmidhuber, 1997; Shi et al., 2015).”
- Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Previous work considers future localization [113, 120, 174, 230], action anticipation [76, 77, 86, 118, 127, 219], next active object prediction [20, 74], future event prediction [149,167], and future frame prediction [145,146,153,215,218,227].”
- Dynamic Filter Networks (2016, methodology): “ding. Several papers have already addressed the generation of an image conditioned on given image(s). Yim et al. [25] and Yang et al. [24] learn to rotate a given face to another pose. The authors of [18, 21, 20, 17, 14] train a deep neural network to predict subsequent video frames. Flynn et al. [5”
- Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (2018, methodology): “We build the shape generator Gmask using a convolutional recurrent neural network [25], as illustrated in Figure 2.”
- Delving Deeper into Convolutional Networks for Learning Video Representations (2015, background): “Recently, Shi et al. (2015) also proposed to leverage convolutional units inside an RNN network.”
- CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, methodology): “We apply the same approach to our program-based video reasoning task ( IEP (V) ) by substituting the program primitives by the ones from CLEVRER, and applying the execution modules on the video features extracted by a convolutional LSTM (Shi et al., 2015).”
- Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search (2018, methodology): “ining it using a previously-proposed distributed RL algorithm (Espeholt et al., 2018). The policy is parameterized as a deep, recurrent neural network consisting of a 3-layer deep convolutional LSTM (Xingjian et al., 2015) with 32 channels per layer and kernel size of 3. To further increase computat”
- Towards Interpretable Reinforcement Learning Using Attention Augmented Agents (2019, background): “The vision core is a multi-layer convolutional network vis✓ followed by a recurrent layer with state svis(t) such as a ConvLSTM [4], which produces an output tensor Ovis 2 Rh⇥w⇥c: Ovis, svis(t) = vis✓(X(t), svis(t 1)) (1) The vision core output is then split along the channel dimension into two tens”
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, methodology): “This includes the Convolutional LSTM (Shi et al., 2015), which replaces the fully-connected layers in an LSTM with convolutional layers to allow for additional structure in the recurrent layers; the Quasi-RNN model (Bradbury et al., 2017) that interleaves convolutional layers with simple recurrent l”
- Feature Boosting Network For 3D Pose Estimation (2019, methodology): “STM) [36], a variant of the original LSTM that can handle 2D input data, as the main building block of our LSTD module.”
- Unsupervised Learning of Long-Term Motion Dynamics for Videos (2017, methodology): “epresentation (fixing the weights). Then, a classification layer is trained to infer the activities. preserve the spatial information in intermediate representations, we use the convolutional LSTM unit [41] that has convolutional structures in both the input-to-state and stateto-state transitions. Her”