Learning Temporal Transformations from Time-Lapse Videos

By Yipin Zhou, Tamara L. Berg. 2016

148 citations recorded by the source corpus.

Based on life-long observations of physical, chemical, and biologic phenomena in the natural world, humans can often easily picture in their minds what an object will look like in the future. But, what about computers? In this paper, we learn computational models of object transformations from time-lapse videos. In particular, we explore the use of generative models to create depictions of objects at future times. These models explore several different prediction tasks: generating a future state given a single depiction of an object, generating a future state given two depictions of an object at different times, and generating future states recursively in a recurrent framework. We provide both qualitative and quantitative evaluations of the generated results, and also conduct a human evaluation to compare variations of our models.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Generating Videos with Scene Dynamics (2016, background): “Often these works may be viewed as a generative model conditioned on the past frames.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, neutral): “Relation to existing tasks Limited prior work considers object state change in photos [102,164] or video [8,68,242]; Ego4D is the first video benchmark dedicated to the task of understanding object state changes.”
Mode Regularized Generative Adversarial Networks (2016, methodology): “…image synthesis from text Reed et al. (2016) and edge map Isola et al. (2016), real-time image manipulation Zhu et al. (2016), temporal image generation Zhou & Berg (2016); Saito & Matsumoto (2016); Vondrick et al. (2016), texture synthesis, style transfer, and video stylization Li & Wand (2016).”
Maximum-Likelihood Augmented Discrete Generative Adversarial Networks (2017, background): “…Zhang et al., 2016b) and unconditionally (Radford et al., 2015; Nguyen et al., 2016), to realize image manipulation and super-resolution (Zhu et al., 2016; Sønderby et al., 2017; Ledig et al., 2016), and to produce video sequences (Mathieu et al., 2016; Zhou & Berg, 2016; Saito & Matsumoto, 2016).”
Stochastic Adversarial Video Prediction (2018, background): “A number of prior works have used adversarial losses for deterministic video prediction [7,11,12,13,40,41].”
Temporal Generative Adversarial Nets with Singular Value Clipping (2016, background): “Zhou and Berg [50] proposed a network that creates depictions of objects at future times with LSTMs and DCGAN.”
Predicting Deeper into the Future of Semantic Segmentation (2017, methodology): “A similar training strategy was employed for future frame predictions in timelapse videos [47].”
A Review on Deep Learning Techniques for Video Prediction (2020, unknown): “long-term dependencies from time-lapse videos [103].”
Every Smile is Unique: Landmark-Guided Diverse Smile Generation (2018, background): “Fostered by the success in image generation, recent works have started to explore deep networks to generate videos [53, 41, 32, 29].”
Adversarial PoseNet: A Structure-Aware Convolutional Network for Human Pose Estimation (2017, background): “The imageconditional models have tackled inpainting [21], image prediction from a normal map [33], future frame prediction [18], future state prediction [40], product photo generation [38], and style transfer [17].”
Learning Dynamic Generator Model by Alternating Back-Propagation Through Time (2018, methodology): “We conduct these experiments on some videos selected from categories “blooming” and “melting” in the dataset of (Zhou and Berg 2016).”
Feedback Adversarial Learning: Spatial Feedback for Improving Generative Adversarial Networks (2019, background): “GANs have continued to show impressive results in various applications such as image generation [46, 15, 63, 8], text to image synthesis [47, 26], future frame prediction [40, 55], image editing [67], novel view synthesis [52, 43], domain adaptation [7, 51], 3D modeling [59, 27], video generation [6”