Transformation-Based Models of Video Sequences

By Joost R. van Amersfoort, Anitha Kannan, Marc'Aurelio Ranzato, Arthur Szlam, Du Tran, Soumith Chintala. 2017

77 citations recorded by the source corpus.

In this work we propose a simple unsupervised approach for next frame prediction in video. Instead of directly predicting the pixels in a frame given past frames, we predict the transformations needed for generating the next frame in a sequence, given the transformations of the past frames. This leads to sharper results, while using a smaller prediction model. In order to enable a fair comparison between different video frame prediction models, we also propose a new evaluation protocol. We use generated frames as input to a classifier trained with ground truth sequences. This criterion guarantees that models scoring high are those producing sequences which preserve discrim- inative features, as opposed to merely penalizing any deviation, plausible or not, from the ground truth. Our proposed approach compares favourably against more sophisticated ones on the UCF-101 data set, while also being more efficient in terms of the number of parameters and computational cost.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[215] Joost Van Amersfoort, Anitha Kannan, Marc’Aurelio Ranzato, Arthur Szlam, Du Tran, and Soumith Chintala.”
MoCoGAN: Decomposing Motion and Content for Video Generation (2017, background): “The future frame prediction problem studied in [34, 26, 24, 17, 10, 38, 44, 40, 7] is different to the video generation problem.”
Stochastic Adversarial Video Prediction (2018, neutral): “An alternative to generating pixels is transforming them by applying a constrained geometric distortion to a previous frame [3,28,29,30,11,31,32,16,13,33,34,35].”
Learning to Decompose and Disentangle Representations for Video Prediction (2018, background): “Instead of directly generating the pixels, transformation-based models focus on predicting the difference/transformation between frames and lead to sharper results [5, 8, 21, 35, 39, 40, 44, 45].”
Deep learning for physical processes: incorporating prior scientific knowledge (2017, result): “[Finn et al., 2016] learn affine transformations on image parts in order to predict object displacement and [Van Amersfoort et al., 2017] proposed a similar model.”
A Review on Deep Learning Techniques for Video Prediction (2020, background): “[71] 2017 [77] CNN [74], [115] RGB RGB X MSE SR ? ? 7 7”
Transformation-based Adversarial Video Prediction on Large-Scale Data (2020, background): “Kernel-based approaches predict the parameters of dynamic, depth-wise locally connected layers [2, 80, 21, 69, 55].”
SimVP: Simpler yet Better Video Prediction (2022, unknown): “2014-2015 [41, 55, 71] [36, 46] [40] 2016-2017 [38, 47, 63, 67] [3, 9, 15, 32, 62] [20, 35, 59] 2018-2019 [22, 43, 56, 65, 69, 74] [6, 10, 61, 66, 73] [70] [16, 29, 72] 2020-2021 [68] [18, 23] [48] [7, 54]”
Animating Arbitrary Objects via Deep Motion Transfer (2018, background): “Video generation is closely related to the future frame prediction problem addressed in [34, 26, 13, 40, 48].”
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, background): “DIGAN results show periodic artifacts induced by the sinusoidal positional encodings (repeated changes in the diagonal direction).”
First Order Motion Model for Image Animation (2020, background): “Another typical case of conditional generation is the problem of future frame prediction, in which the generated video is conditioned on the initial frame [12, 23, 30, 35, 44].”
Predicting Deeper into the Future of Semantic Segmentation (2017, unknown): “To reduce the number of parameters to estimate, several authors reparameterize the problem to predict frame transformations instead of raw pixels [11, 37].”
Future Frame Prediction for Anomaly Detection - A New Baseline (2017, background): “Different from these work, recently, people propose to predict transformations needed for generating future frames [33] and [4], which further boosts the performance of video prediction.”
DeepLandscape: Adversarial Modeling of Landscape Video (2020, methodology): “Our approach is thus opposed to methods that animate landscapes and textures by generating warping ﬁelds applied to the raw pixels of the input static image [2, 19, 11, 20, 21].”