Spatio-temporal video autoencoder with differentiable memory

By Viorica Patraucean, Ankur Handa, Roberto Cipolla. 2015

312 citations recorded by the source corpus.

We describe a new spatio-temporal video autoencoder, based on a classic spatial image autoencoder and a novel nested temporal autoencoder. The temporal encoder is represented by a differentiable visual memory composed of convolutional long short-term memory (LSTM) cells that integrate changes over time. Here we target motion changes and use as temporal decoder a robust optical flow prediction module together with an image sampler serving as built-in feedback loop. The architecture is end-to-end differentiable. At each time step, the system receives as input a video frame, predicts the optical flow based on the current observation and the LSTM memory state as a dense transformation map, and applies it to the current frame to generate the next frame. By minimising the reconstruction error between the predicted next frame and the corresponding ground truth next frame, we train the whole system to extract features useful for motion estimation without any supervision effort. We present one direct application of the proposed framework in weakly-supervised semantic segmentation of videos through label propagation using optical flow.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Scaling Autoregressive Video Models (2019, background): “, 2016b; Denton & Fergus, 2018) or generation of optical flow (Ptrucean et al., 2016).”
Video Pixel Networks (2016, background): “We show that the VPN not only generalizes to new action sequences with objects seen during training, but also to new action sequences involving novel objects not seen during training.”
Dynamic Filter Networks (2016, background): “Patraucean et al. [17] extend the Spatial Transformer by modifying the grid generator such that it has one transformation for each position, instead of a single transformation for the entire image.”
Recurrent Environment Simulators (2017, background): “Deep neural networks have recently enabled significant advances in simulating complex environments, allowing for models that consider high-dimensional visual inputs across a wide variety of domains (Wahlström et al., 2015; Watter et al., 2015; Sun et al., 2015; Patraucean et al., 2015).”
A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning (2017, background): “Instead of basing inference on any abstract grasp of dynamics that is learned from experience, current successes are autoregressive: to imagine the tennis ball’s trajectory, one forward-generates a frame-by-frame rendering of the full sensory input [5, 7, 23, 24, 29, 30].”
Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning (2016, background): “Here, we explore another potential principle for exploiting video for unsupervised learning: prediction of future image frames [16, 28, 32, 33, 34, 46, 48].”
Learning to Decompose and Disentangle Representations for Video Prediction (2018, methodology): “For reference, we also list the results of existing work on Moving MNIST, where they use more complicated models such as convolutional LSTM or PixelCNN decoders [15, 25, 43].”
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (2022, methodology): “[49] Viorica Patraucean, Ankur Handa, and Roberto Cipolla.”
A Review on Deep Learning Techniques for Video Prediction (2020, background): “FSTN [78] 2017 [44], [77] LSTM-cED [74], [115], [123], [131], [132] RGB RGB X `2, `δ, PL SR ? ? ? 7 7”
Learning Video Object Segmentation with Visual Memory (2017, methodology): “The other variant using ConvLSTM is comparable to ConvRNN, possibly due to the lack of suf-ﬁcient training data.”
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation (2017, background): “Instead of merely considering two frames [38], a memory module is proposed to keep the temporal information of a video sequence [21].”
PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning (2021, background): “the visual appearances and temporal dynamics, and is also a foundation for the follow-up approaches [4], [9], [25], [26], [27], [28], [29], [65], [66], [67], [68], [69], [70], [71].”
Self-supervised learning of a facial attribute embedding from video (2018, unknown): “work trained to recognise the rotation [19] or to learn equivariant pixel embeddings [40,51,52]. Of more direct relevance to our training framework are self-supervised frameworks that use video data ([1,8,11,16,17,21,22,30,34,44,55,56,58,61]). Our approach builds in particular on those that use fram”
Bidirectional Multirate Reconstruction for Temporal Modeling in Videos (2016, methodology): “[31] used a spatio-temporal autoencoder to model video sequences through optical flow prediction and reconstruction of the next frame.”
Unsupervised Learning of Object Landmarks through Conditional Image Generation (2018, unknown): “Several other generative methods [42, 40, 37, 48, 32] focus on video extrapolation.”
Long-Term Image Boundary Prediction (2016, methodology): “We report quantitative results for prediction one time-step into the future (as in (Srivastava, Mansimov, and Salakhutdinov 2015; Patraucean, Handa, and Cipolla 2015)) in Table 7 using the Cross Entropy Loss (Srivastava, Mansimov, and Salakhutdinov 2015).”
A review of convolutional neural networks in computer vision (2024, unknown): “ConvLSTM and an optical flow predictor are used in the spatio-temporal video auto-encoder (Patraucean et al. 2015) to record changes over time.”
Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation (2016, background): “It has been shown how spatial transformers can effectively encode optical ﬂow to describe motion [29, 1, 14], and are therefore suitable for motion compensation.”
Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction (2020, methodology): “To constrain the challenging generation of high dimensional images, several methods rather predict geometric transformations between frames [16, 24, 75] or use optical flow [46, 38, 33, 32, 31].”