PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning

By Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, Philip S. Yu, Mingsheng Long. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021

647 citations recorded by the source corpus.

The predictive learning of spatiotemporal sequences aims to generate future images by learning from the historical context, where the visual dynamics are believed to have modular structures that can be learned with compositional subsystems. This paper models these structures by presenting PredRNN, a new recurrent network, in which a pair of memory cells are explicitly decoupled, operate in nearly independent transition manners, and finally form unified representations of the complex environment. Concretely, besides the original memory cell of LSTM, this network is featured by a zigzag memory flow that propagates in both bottom-up and top-down directions across all layers, enabling the learned visual dynamics at different levels of RNNs to communicate. It also leverages a memory decoupling loss to keep the memory cells from learning redundant features. We further propose a new curriculum learning strategy to force PredRNN to learn long-term dynamics from context frames, which can be generalized to most sequence-to-sequence models. We provide detailed ablation studies to verify the effectiveness of each component. Our approach is shown to obtain highly competitive results on five datasets for both action-free and action-conditioned predictive learning scenarios.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d8041b594e52d1ef27288c7fd5ecd2f846d1c768

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dit-Yan Yeung
2. Hao Wang
3. Limin Wang
4. Yufei Zhang
5. Mu Li
6. Yi Zhu
7. Xia Zhao
8. Yuyang Wang

Citations in the corpus

• Earthformer: Exploring Space-Time Transformers for Earth System Forecasting (2022, methodology): “Conventional DL models for Earth system forecasting are based on CNN and RNN. U-Net with either 2D CNN or 3D CNN have been used for precipitation nowcasting [43], Seasonal Arctic Sea ice prediction [1], and ENSO forecasting [15].”
• SimVP: Simpler yet Better Video Prediction (2022, neutral): “867 PyTorch PredRNN-V2 [68] (Arxiv 2021) 48.”
• A review of convolutional neural networks in computer vision (2024, positive): “In response to the complexity and future uncertainty of video itself, scholars have achieved impressive results in a video in recent years by introducing various new neural operators, such as various RNNs (Wang et al. 2022), transformers (Ren et al. 2022), refinement structures (Chang et al. 2022), ”