Sync-DRAW: Automatic Video Generation using Deep Recurrent Attentive Architectures

By Gaurav Mittal, Tanya Marwah, Vineeth Nallure Balasubramanian. 2016

79 citations recorded by the source corpus.

This paper introduces a novel approach for generating videos called Synchronized Deep Recurrent Attentive Writer (Sync-DRAW). Sync-DRAW can also perform text-to-video generation which, to the best of our knowledge, makes it the first approach of its kind. It combines a Variational Autoencoder(VAE) with a Recurrent Attention Mechanism in a novel manner to create a temporally dependent sequence of frames that are gradually formed over time. The recurrent attention mechanism in Sync-DRAW attends to each individual frame of the video in sychronization, while the VAE learns a latent distribution for the entire video at the global level. Our experiments with Bouncing MNIST, KTH and UCF-101 suggest that Sync-DRAW is efficient in learning the spatial and temporal information of the videos and generates frames with high structural integrity, and can generate videos from simple captions on these datasets.

Subfields: Artificial Intelligence, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Nan Duan
2. Devi Parikh
3. Sanja Fidler
4. Mohit Bansal
5. Adam Polyak
6. Jianfeng Gao
7. Huan Ling
8. Fan Yang

In Generative Models

1. Andreas Blattmann
2. Karsten Kreis
3. Tim Dockhorn
4. Robin Rombach
5. Huan Ling
6. Sanja Fidler
7. Seung Wook Kim
8. Adam Polyak

Citations in the corpus

• VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “, 2015), variational autoencoders (VAEs) (Kingma & Welling, 2013; Kingma et al., 2016; Mittal et al., 2017; Marwah et al., 2017; Vahdat & Kautz, 2020; Child, 2020), Flows (Dinh et al.”
• Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, background): “[55] Gaurav Mittal, Tanya Marwah, and Vineeth N.”
• Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, methodology): “SyncDraw [44] first proposes to combine VAE and RNN for video generation while conditioning on texts.”
• Make-A-Video: Text-to-Video Generation without Text-Video Data (2022, methodology): “To our knowledge, Sync-DRAW (Mittal et al., 2017) is the first T2V generation approach that leverages a VAE with recurrent attention.”
• Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, unknown): “Early works [28, 33, 26, 24, 12, 25] primarily focus on generating videos in simple domains, such as moving digits or specific human actions.”
• GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions (2021, methodology): “VGAN[27] SyncDraw[14] TGANs[15] MocoGAN[25] IRC-GAN[6] GODIVA(ours) Figure 4: Comparison of samples from our model to prior approaches on Moving Mnist and Double Moving Mnist dataset.”