StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2

By Ivan Skorokhodov, Sergey Tulyakov, Mohamed Elhoseiny. Computer Vision and Pattern Recognition, 2021

390 citations recorded by the source corpus.

Videos show continuous events, yet most - if not all - video synthesis frameworks treat them discretely in time. In this work, we think of videos of what they should be - time-continuous signals, and extend the paradigm of neural representations to build a continuous-time video generator. For this, we first design continuous motion representations through the lens of positional embeddings. Then, we explore the question of training on very sparse videos and demon-strate that a good generator can be learned by using as few as 2 frames per clip. After that, we rethink the traditional image + video discriminators pair and design a holistic dis-criminator that aggregates temporal information by simply concatenating frames' features. This decreases the training cost and provides richer learning signal to the generator, making it possible to train directly on 10242 videos for the first time. We build our model on top of StyleGAN2 and it is just ≈5% more expensive to train at the same resolution while achieving almost the same image quality. Moreover, our latent space features similar properties, enabling spa-tial manipulations that our method can propagate in time. We can generate arbitrarily long videos at arbitrary high frame rate, while prior work struggles to generate even 64 frames at a fixed rate. Our model is tested on four mod-ern 2562 and one 10242 -resolution video synthesis bench-marks. In terms of sheer metrics, it performs on average ≈30% better than the closest runner-up. Project website: https://universome.github.io/stylegan-v.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d97e0adbade91d76b10e8790205a71877a9be42b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “The Bund, Shanghai, with the ship moving on the river, oil painting.”
MotionCtrl: A Unified and Flexible Motion Controller for Video Generation (2023, neutral): “Video generation, especially text-to-video (T2V) generation strives to create varied and high-fidelity videos that align with specified textual descriptions.”
Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (2023, methodology): “Various generation methods have been proposed in the past, including VAE-based (Variational Autoen-coder) [16, 33, 58, 61], GAN-based (generative adversarial networks) [3, 15, 31, 49, 56, 70], flow-based [10, 34] and auto-regressive models [18, 64, 68].”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, methodology): “Previously studies have resorted to different types of generative models such as GANs [58, 43, 54, 52, 45], Autoregressive models [51, 63, 25, 9, 18], and implicit neural representations [47, 66].”
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, methodology): “StyleGAN-v [61] maps a continuous temporal positional embedding to the input feature map of the StyleGAN.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, methodology): “GAN-based methods produce more realistic videos [59, 60, 61, 62, 63, 64] but are known to suffer from training instability and a lack of generation diversity [65].”
ModelScope Text-to-Video Technical Report (2023, background): “While video generation [52, 20, 67] continues to pose challenges.”
Latent Video Diffusion Models for High-Fidelity Long Video Generation (2022, methodology): “Existing works capitalize on different types of generative models including GANs [40, 26, 27, 36, 35, 30, 2], VAEs [7, 44], autoregressive models [6, 42, 21], and normalizing ﬂows [17].”
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, background): “However, the quality of the generated videos is often low quality or can only work on a specific domain, e.g ., face [66] or landscape [50,63].”
MAGVIT: Masked Generative Video Transformer (2022, neutral): “Early success in video synthesis has been made by GAN models [1,6,7,15,27,37,48,51, 56, 59, 66, 74].”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, methodology): “We construct a benchmark of unconditional video generation task, for four currently prevalent models (VideoGPT [90], MoCoGAN-HD [75], DIGAN [94], and StyleGAN-V [73]) on 4 face video datasets (FaceForensics [65], Vox [59], MEAD [82] and CelebV-HQ).”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, neutral): “…2017; Tulyakov et al., 2018; Tian et al., 2021; Shen et al., 2023), autoregressive models (Srivastava et al., 2015; Yan et al., 2021; Le Moing et al., 2021; Ge et al., 2022; Hong et al., 2022; Kondratyuk et al., 2023), and implicit neural representations (Skorokhodov et al., 2021; Yu et al., 2021).”
I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models (2023, background): “Early research in video generation [16, 40, 54, 59] primarily focuses on using GAN-related methods, but maintaining spatio-temporal coherence as well as realism remains a significant challenge.”
Pix2Video: Video Editing using Image Diffusion (2023, neutral): “Until recently, GANs have been the method of choice for video generation, with many works designed towards unconditional generation [5, 13, 45, 47, 51, 62, 64].”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, background): “[78] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny.”
Structure and Content-Guided Video Synthesis with Diffusion Models (2023, background): “These methods often struggle with stability during optimization, and produce fixed-length videos [59, 45] or longer videos where artifacts accumulate over time [50].”
Latte: Latent Diffusion Transformer for Video Generation (2024, neutral): “Following the experimental setup in (Skorokhodov et al, 2022), except for UCF101, we use the training split for all datasets if they are available.”