Video Generation From Text

By Yitong Li, Martin Renqiang Min, Dinghan Shen, David Edwin Carlson, Lawrence Carin. AAAI Conference on Artificial Intelligence, 2017

317 citations recorded by the source corpus.

Generating videos from text has proven to be a significant challenge for existing generative models. We tackle this problem by training a conditional generative model to extract both static and dynamic information from text. This is manifested in a hybrid framework, employing a Variational Autoencoder (VAE) and a Generative Adversarial Network (GAN). The static features, called "gist," are used to sketch text-conditioned background color and object layout structure. Dynamic features are considered by transforming input text into an image filter. To obtain a large amount of data for training the deep-learning model, we develop a method to automatically create a matched text-video corpus from publicly available online videos. Experimental results show that the proposed framework generates plausible and diverse short-duration smooth videos, while accurately reflecting the input text information. It significantly outperforms baseline models that directly adapt text-to-image generation procedures to produce videos. Performance is evaluated both visually and by adapting the inception score used to evaluate image generation in GANs.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/3a2bbf1f895a1850da3bb6d92b4ffbe61b68145d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Perceptual Pyramid Adversarial Networks for Text-to-Image Synthesis (2019, methodology): “Recently, GANs have been applied to various applications, especially in simulating complicated data distributions, like images (Song et al. 2018a; 2018b), music, texts (Gupta et al. 2018) and video (Li et al. 2018).”
Make-A-Video: Text-to-Video Generation without Text-Video Data (2022, background): “, 2017) and (Li et al., 2018) extend GANs from image generation to T2V generation.”
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, methodology): “T2V [38] uses CVAE to generate the gist then GANs with 3D convolutions to generate fixed-length low-resolution videos.”
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, unknown): “Early works [27, 32, 25, 23, 11, 24] primarily focus on generating videos in simple domains, such as moving digits or specific human actions.”
Predicting Video with VQVAE (2021, background): “Finally, some have proposed a hierarchical approach based on structured information—generating video conditioned on text [25], semantic segments [29, 28], or human pose [56, 53].”
Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021, background): “The second group is based on generative methods to directly predict the missing modalities given the available modalities (Li et al., 2018; Cai et al., 2018; Suo et al., 2019; Du et al., 2018).”
GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions (2021, methodology): “T2V [11] (64 × 64) Play golf on grass.”
Adversarial Text-to-Image Synthesis: A Review (2021, background): “Similarly, generating videos from textual descriptions seems like an obvious future research direction [184, 185].”
StoryGAN: A Sequential Conditional GAN for Story Visualization (2018, methodology): “To this end, researchers disentangle dynamic and static features for motion and background, respectively [32, 24, 31, 6].”
Generating Diverse and Natural 3D Human Motions from Text (2022, unknown): “To address such task, GAN frameworks have been recruited in several efforts including [20] and [25].”
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion (2021, unknown): “Model Acc↑ FID-img↓ FID-vid↓ CLIPSIM↑ T2V (64×64) [15] 42.”
Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring (2020, methodology): “[10] uses input text to construct the motion-generating filter weights for video generation task.”
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities (2018, unknown): “Even though translation and generation of videos, audios, and text are difficult (Li et al. 2017b), our experiments show that the learned joint representations can help for discriminative tasks: MCTN achieves new state-of-the-art results on multimodal sentiment analysis using the CMUMOSI (Zadeh et a”
Stochastic Variational Video Prediction (2017, background): “Another approach for stochastic prediction uses generative adversarial networks (GANs) (Goodfel-low et al., 2014), which have been used for video generation and prediction (Tulyakov et al., 2017; Li et al., 2017).”
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss (2019, background): “Modeling the dynamics of a moving human face/body conditioned on another modality is a fundamental problem in computer vision, where applications are ranging from audio-to-video generation [28, 3, 2] to text-to-video generation [23, 19] and to skeleton-to-image/video generation [21, 7].”