Attentive Semantic Video Generation Using Captions

By Tanya Marwah, Gaurav Mittal, Vineeth Nallure Balasubramanian. 2017

77 citations recorded by the source corpus.

This paper proposes a network architecture to perform variable length semantic video generation using captions. We adopt a new perspective towards video generation where we allow the captions to be combined with the long-term and short-term dependencies between video frames and thus generate a video in an incremental manner. Our experiments demonstrate our network architecture’s ability to distinguish between objects, actions and interactions in a video and combine them to generate videos for unseen captions. The network also exhibits the capability to perform spatio-temporal style transfer when asked to generate videos for a sequence of captions. We also show that the network’s ability to learn a latent representation allows it generate videos in an unsupervised manner and perform other tasks such as action recognition.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Kristen Grauman
2. Luc Van Gool
3. Dacheng Tao
4. Xiaogang Wang
5. Andrea Vedaldi
6. Trevor Darrell
7. Yi Yang
8. Chenliang Xu

In Generative Models

1. Oron Ashual
2. Sen Wang
3. Lele Chen
4. Devi Parikh
5. Xiaoyue Yin
6. Chenliang Xu
7. Chuan Guo
8. Sonal Gupta

In Multimodal Learning

1. Chenliang Xu
2. Oron Ashual
3. Devi Parikh
4. Yaniv Taigman
5. Zhiyao Duan
6. Adam Polyak
7. Uriel Singer
8. Dhruv Batra

Citations in the corpus

• VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “, 2015), variational autoencoders (VAEs) (Kingma & Welling, 2013; Kingma et al., 2016; Mittal et al., 2017; Marwah et al., 2017; Vahdat & Kautz, 2020; Child, 2020), Flows (Dinh et al.”
• Generating Diverse and Natural 3D Human Motions from Text (2022, methodology): “Moreover, both short-time and long-term cross-domain attentive vectors are utilized in [24] as the inputs to a VAE framework.”
• Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss (2019, background): “Modeling the dynamics of a moving human face/body conditioned on another modality is a fundamental problem in computer vision, where applications are ranging from audio-to-video generation [28, 3, 2] to text-to-video generation [23, 19] and to skeleton-to-image/video generation [21, 7].”
• Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation (2022, unknown): “Early works [28, 33, 26, 24, 12, 25] primarily focus on generating videos in simple domains, such as moving digits or specific human actions.”
• Make-A-Video: Text-to-Video Generation without Text-Video Data (2022, methodology): “To our knowledge, Sync-DRAW (Mittal et al., 2017) is the first T2V generation approach that leverages a VAE with recurrent attention.”