PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning

By Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, Philip S. Yu. International Conference on Machine Learning, 2018

613 citations recorded by the source corpus.

We present PredRNN++, an improved recurrent network for video predictive learning. In pursuit of a greater spatiotemporal modeling capability, our approach increases the transition depth between adjacent states by leveraging a novel recurrent unit, which is named Causal LSTM for re-organizing the spatial and temporal memories in a cascaded mechanism. However, there is still a dilemma in video predictive learning: increasingly deep-in-time models have been designed for capturing complex variations, while introducing more difficulties in the gradient back-propagation. To alleviate this undesirable effect, we propose a Gradient Highway architecture, which provides alternative shorter routes for gradient flows from outputs back to long-range inputs. This architecture works seamlessly with causal LSTMs, enabling PredRNN++ to capture short-term and long-term dependencies adaptively. We assess our model on both synthetic and real video datasets, showing its ability to ease the vanishing gradient problem and yield state-of-the-art prediction results even in a difficult objects occlusion scenario.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/d718941506d2adabc4792cb13d49e6336957e52e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Review on Deep Learning Techniques for Video Prediction (2020, methodology): “Alternatives to RNNs: Currently, RNNs are still widely used in this field to model temporal dependencies, and achieved state-of-the-art results on different benchmarks [66], [153], [197], [235].”
SimVP: Simpler yet Better Video Prediction (2022, methodology): “Sixteen base-lines are included, such as MCnet [62], ConvLSTM [71], SAVP, SAVP-VAE [30], VPN [28], DFN [25], fRNN [43], Znet [74], SV2P [3], PredRNN [67], VarNet [27], Pre-dRNN++ [65], MSNET [31], E3d-LSTM [66] and STM-FANet [26].”
Unsupervised Learning From Video With Deep Neural Embeddings (2019, background): “Temporal autoencoders such as PredNet [22], PredRNN [35], and PredRNN++ [34] are intriguing but have not yet evidenced substantial transfer learning performance at scale.”
Memory in Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity From Spatiotemporal Dynamics (2018, methodology): “In particular, we construct another model named MIM* by using Causal LSTM [31] as the ﬁrst layer, and integrating the cascaded MIM-N and MIM-S modules into the Causal LSTM memory cells, using them to replace the temporal forget gates in Causal LSTMs.”
Latent Video Transformer (2020, neutral): “Video prediction and unconditional video synthesis have been addressed for a long time and the solutions include recurrent models [15, 47, 48, 4], VAE-based models [8, 11, 25, 20, 9], autoregressive models [33, 39, 50, 22, 49, 19], normalizing ﬂows [24], GANs [27, 44, 35, 40, 1, 36, 7, 26] and…”
Self-Attention ConvLSTM for Spatiotemporal Prediction (2020, methodology): “ConvLSTM (Shi et al. 2015) replaces all the linear operations in it with convolution layers to capture spatial dependencies besides the long-short term modeling, and many of its variants (Wang et al. 2017a; 2018b; 2019) have achieved impressive results on spatiotemporal prediction.”
Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction (2020, methodology): “This is similar to gating mechanisms in LSTMs or GRUs.”
PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning (2021, methodology): “However, the two components mentioned above in PredRNN++ introduce additional parameters and increase the computational burden.”