MoCoGAN: Decomposing Motion and Content for Video Generation

By Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, Jan Kautz. 2017

1,315 citations recorded by the source corpus.

Visual signals in a video can be divided into content and motion. While content specifies which objects are in the video, motion describes their dynamics. Based on this prior, we propose the Motion and Content decomposed Generative Adversarial Network (MoCoGAN) framework for video generation. The proposed framework generates a video by mapping a sequence of random vectors to a sequence of video frames. Each random vector consists of a content part and a motion part. While the content part is kept fixed, the motion part is realized as a stochastic process. To learn motion and content decomposition in an unsupervised manner, we introduce a novel adversarial learning scheme utilizing both image and video discriminators. Extensive experimental results on several challenging datasets with qualitative and quantitative comparison to the state-of-the-art approaches, verify effectiveness of the proposed framework. In addition, we show that MoCoGAN allows one to generate videos with same content but different motion as well as videos with different content and same motion. Our code is available at https://github.com/sergeytulyakov/mocogan.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

VideoBERT: A Joint Model for Video and Language Representation Learning (2019, methodology): “There are also various GAN-based approaches, such as the SAVP approach of [13] and the MoCoGAN approach of [27].”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “MoCoGAN [48] proposed the motion and content decomposed generative adversarial networks for video generation.”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “While there have certainly been impressive efforts to model videos (Vondrick et al., 2016; Kalchbrenner et al., 2016; Tulyakov et al., 2018; Clark et al., 2019), high-fidelity natural videos is one notable modality that has not seen the same level of progress in generative modeling as compared to im”
Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, background): “Generative models [28], [29] are widely used in many areas for its strong ability of modeling and learning approximately true dataset distributions, including image generation [30]–[32], representation learning [33], image retrieval [34], [35], object detection [36], video applications [37], [38] an”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “GANs enable a wide variety of applications such as image generation [1, 41, 60], representation learning [44], image manipulation [62], object detection [32], and video applications [37, 50, 52].”
Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, background): “For video generation, Tulyakov et al. [58] proposed a MoCoGAN to generate video in an unsupervised manner.”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, methodology): “There are also various GAN-based approaches, such as the SAVP approach of Lee et al. (2018) and the MoCoGAN approach of Tulyakov et al. (2018).”
Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects (2018, background): “Another interesting line of work is the GAN -based unsupervised video generation that decomposes motion and content (Tulyakov et al., 2018; Denton and Birodkar, 2017).”
Stochastic Variational Video Prediction (2017, methodology): “, 2014), which have been used for video generation and prediction (Tulyakov et al., 2017; Li et al., 2017).”
Alias-Free Generative Adversarial Networks (2021, background): “They have been used for a variety of applications, including image editing [47, 53, 42, 21, 38, 3], domain translation [66, 36, 58, 41], and video generation [55, 15, 23].”
Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space? (2019, neutral): “texture synthesis [18, 33, 28], video generation [31, 30], image-to-image translation [11, 36, 1, 24] and object detection [19].”
Stochastic Adversarial Video Prediction (2018, methodology): “This model is broadly representative of prior stochastic GAN-based methods [14,15,16,17].”
Video-to-Video Synthesis (2018, methodology): “The video synthesis problem exists in various forms, including future video prediction [15,18,42,45, 50,65,68,71,77] and unconditional video synthesis [59,67,69].”
Learning to Decompose and Disentangle Representations for Video Prediction (2018, background): “The idea of disentangled representation has already been explored [6, 34, 37] for video.”
Video Diffusion Models (2022, neutral): “Method Resolution FID↓ IS↑ MoCoGAN [52] 16x64x64 26998 ± 33 12.”
Image2StyleGAN++: How to Edit the Embedded Images? (2019, background): “object detection [23], texture synthesis [22, 37, 31], image-to-image translation [16, 42, 28, 25] and video generation [33, 32, 35, 34].”
How Generative Adversarial Networks and Their Variants Work (2017, methodology): “Their capability to represent complex and high-dimensional data can be utilized in treating images [2, 13, 60, 68, 132, 138, 150], videos [127, 130, 131], music generation [43, 69, 146], natural languages [51, 76] and other academic domains such as medical images [17, 80, 141] and security [114, 129”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, methodology): “Later, Motion and Content GAN (MoCo-GAN) [163] introduced a video generation framework.”