Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction
By Vincent Le Guen, Nicolas Thomé. Computer Vision and Pattern Recognition, 2020
426 citations recorded by the source corpus.
Leveraging physical knowledge described by partial differential equations (PDEs) is an appealing way to improve unsupervised video forecasting models. Since physics is too restrictive for describing the full visual content of generic video sequences, we introduce PhyDNet, a two-branch deep architecture, which explicitly disentangles PDE dynamics from unknown complementary information. A second contribution is to propose a new recurrent physical cell (PhyCell), inspired from data assimilation techniques, for performing PDE-constrained prediction in latent space. Extensive experiments conducted on four various datasets show the ability of PhyDNet to outperform state-of-the-art methods. Ablation studies also highlight the important gain brought out by both disentanglement and PDE-constrained prediction. Finally, we show that PhyDNet presents interesting features for dealing with missing data and long-term forecasting.
View Paper: https://www.semanticscholar.org/paper/603a0764092fbda01f3414071ea2813c49e1efa3
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Mingsheng Long
- Hao Wang
- Jianmin Wang
- Dit-Yan Yeung
- Yunbo Wang
- Philip S. Yu
- Xingjian Shi
- Zhangyang Gao
Citations in the corpus
- PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning (2021, background): “[63], and solvers of neural differential equations [64].”
- SimVP: Simpler yet Better Video Prediction (2022, methodology): “Similar to [18, 55, 69], only videos with ”walking” scenario are used, and 4 future frames are generated by feeding previous 4 RGB frames.”
- Can deep learning beat numerical weather prediction? (2021, background): “Le Guen et al. [76] introduced a new two-stream model and a recurrent cell, which is based on concepts from DA.”
- PIP: Physical Interaction Prediction via Mental Imagery with Span Selection (2021, methodology): “We modify PhyDNet [31] for a performance comparison between our approach and the incorporation of physical dynamics in mental simulations.”
- Earthformer: Exploring Space-Time Transformers for Earth System Forecasting (2022, methodology): “Conventional DL models for Earth system forecasting are based on CNN and RNN. U-Net with either 2D CNN or 3D CNN have been used for precipitation nowcasting [43], Seasonal Arctic Sea ice prediction [1], and ENSO forecasting [15].”