Markov Decision Process for Video Generation
By Vladyslav Yushchenko, Nikita Araslanov, Stefan Roth. 2019
21 citations recorded by the source corpus.
We identify two pathological cases of temporal inconsistencies in video generation: video freezing and video looping. To better quantify the temporal diversity, we propose a class of complementary metrics that are effective, easy to implement, data agnostic, and interpretable. Further, we observe that current state-of-the-art models are trained on video samples of fixed length thereby inhibiting long-term modeling. To address this, we reformulate the problem of video generation as a Markov Decision Process (MDP). The underlying idea is to represent motion as a stochastic process with an infinite forecast horizon to overcome the fixed length limitation and to mitigate the presence of temporal artifacts. We show that our formulation is easy to integrate into the state-of-the-art MoCoGAN framework. Our experiments on the Human Actions and UCF-101 datasets demonstrate that our MDP-based model is more memory efficient and improves the video quality both in terms of the new and established metrics.
Subfields: Computer Vision and Pattern Recognition, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Jitendra Malik
- Xiaogang Wang
- Andrew Zisserman
- Chen Change Loy
In Generative Models
- Jonathan Ho
- David J. Fleet
- Mohammad Norouzi
- Tim Salimans
- William Chan
- Pieter Abbeel
- Arash Vahdat
- Stefano Ermon
Citations in the corpus
- VideoGPT: Video Generation using VQ-VAE and Transformers (2021, background): “Other examples of prior work with video generation of GANs include (Saito et al., 2017), (Tulyakov et al., 2018), (Acharya et al., 2018), (Yushchenko et al., 2019).”
- Video Diffusion Models (2022, methodology): “As discussed by [64], methods in the literature are unfortunately not always consistent in the data preprocessing that is used, which may lead to small differences in reported scores between papers.”
- MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation (2022, unknown): “UCF-101 [0→ 16; trained on k] k FVD↓ MoCoGAN-MDP [Yushchenko et al., 2019] 16 1277.”