Temporal Generative Adversarial Nets with Singular Value Clipping

By Masaki Saito, Eiichi Matsumoto, Shunta Saito. 2016

509 citations recorded by the source corpus.

In this paper, we propose a generative model, Temporal Generative Adversarial Nets (TGAN), which can learn a semantic representation of unlabeled videos, and is capable of generating videos. Unlike existing Generative Adversarial Nets (GAN)-based methods that generate videos with a single generator consisting of 3D deconvolutional layers, our model exploits two different types of generators: a temporal generator and an image generator. The temporal generator takes a single latent variable as input and outputs a set of latent variables, each of which corresponds to an image frame in a video. The image generator transforms a set of such latent variables into a video. To deal with instability in training of GAN with such advanced networks, we adopt a recently proposed model, Wasserstein GAN, and propose a novel method to train it stably in an end-to-end manner. The experimental results demonstrate the effectiveness of our methods.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, background): “TGAN [47] exploited a 1D temporal generator and a 2D image generator for video generation.”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “, 2016), TGAN (Saito et al., 2017), MoCoGAN (Tulyakov et al.”
cGANs with Projection Discriminator (2018, background): “…y into the discriminator by naively concatenating (embedded) y to the input or to the feature vector at some middle layer (Mirza & Osindero, 2014; Denton et al., 2015; Reed et al., 2016; Zhang et al., 2017; Perarnau et al., 2016; Saito et al., 2017; Dumoulin et al., 2017a; Sricharan et al., 2017).”
MoCoGAN: Decomposing Motion and Content for Video Generation (2017, methodology): “All values except MoCoGAN’s are taken from [31].”
Stochastic Adversarial Video Prediction (2018, methodology): “This model is broadly representative of prior stochastic GAN-based methods [14,15,16,17].”
Video-to-Video Synthesis (2018, background): “The video synthesis problem exists in various forms, including future video prediction [15,18,42,45, 50,65,68,71,77] and unconditional video synthesis [59,67,69].”
Few-shot Video-to-Video Synthesis (2019, background): “Video generative models can be divided into three main categories, including 1) unconditional video synthesis models [54, 45, 51], which convert random noise samples to video clips, 2) future video prediction models [48, 24, 11, 34, 33, 63, 55, 56, 10, 53, 29, 27, 18, 28, 16, 40], which generate…”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, background): “Conventional unconditional video face generation [80,66,77,94] are based on GANs [24].”
RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces (2020, background): “Several studies have also attempted to learn entire video distributions as spatio-temporal tensors in GAN frameworks [17, 32, 38, 41] yielding impressive first results for full video generation in artificial and real domains.”
Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos (2020, background): “Most of latent-representation based methods [1], [4], [11], [12], [26] encode the background, object pose and appearance into one latent representation.”
Deep Audio-visual Learning: A Survey (2020, methodology): “t of audio input. The other was a conditional GAN (cGAN) used to generate the resulting faces conditioned on a given set of lip landmarks. Instead of applying cGAN, [98] proposed using a temporal GAN [111] to improve the quality of synthesis. However, the above methods were only applicable to synthe”
Structure and Content-Guided Video Synthesis with Diffusion Models (2023, background): “These methods often struggle with stability during optimization, and produce fixed-length videos [59, 45] or longer videos where artifacts accumulate over time [50].”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, background): “[70] Masaki Saito, Eiichi Matsumoto, and Shunta Saito.”