Unsupervised Learning of Long-Term Motion Dynamics for Videos

By Zelun Luo, Boya Peng, De-An Huang, Alexandre Alahi, Li Fei-Fei. 2017

196 citations recorded by the source corpus.

We present an unsupervised representation learning approach that compactly encodes the motion dependencies in videos. Given a pair of images from a video clip, our framework learns to predict the long-term 3D motions. To reduce the complexity of the learning framework, we propose to describe the motion as a sequence of atomic 3D flows computed with RGB-D modality. We use a Recurrent Neural Network based Encoder-Decoder framework to predict these sequences of flows. We argue that in order for the decoder to reconstruct these sequences, the encoder must learn a robust video representation that captures long-term motion dependencies and spatial-temporal relations. We demonstrate the effectiveness of our learned temporal representations on activity classification across multiple modalities and datasets such as NTU RGB+D and MSR Daily Activity 3D. Our framework is generic to any input modality, i.e., RGB, depth, and RGB-D videos.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Contrastive Multiview Coding (2019, neutral): “1 LT-Motion* [46] 2 53.”
NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, methodology): “Speciﬁcally, many of them have been evaluated based on the preliminary version [47] of our dataset, or pre-trained on it for transfer learning for other tasks [43], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79].”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “Most of available datasets require subjects to perform actions in a visible and restricted view to avoid occlusion, and this results in limited view data collection and less 29 SBU Kinect Interaction [149] Depth+ Skeleton PRNN Side Information 89.20% [130] Skeleton Co-occurrence +LSTM None 90.41% [1”
Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates (2017, neutral): “[25], [26], [27], [28], [29], [30], and RGB-based activity recognition [31], [32], [33], [34], [35], [36].”
Transitive Invariance for Self-Supervised Visual Representation Learning (2017, background): “v:1708.02901v3 [cs.CV] 15 Aug 2017 carefully annotated data. However, large-scale, high-quality annotations come at a cost of expensive human effort. Unsupervised or “self-supervised” learning (e.g., [61,9, 45,63,64,35,44,62,40,66]) recently has attracted increasing interests because the “labels” ar”
Cross and Learn: Cross-Modal Self-Supervision (2018, methodology): “LT-Motion [22] (RNN) NTU (57K videos) 53.”
Label Efficient Learning of Transferable Representations acrosss Domains and Tasks (2017, background): “An alternative approach, often terms “self-supervised learning” [5], defines a pretext task such as predicting patch ordering [7], frame ordering [40], motion dynamics [39], or colorization [72], as a form of indirect supervision.”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, unknown): “Mehtod SubCategory Code Contribution VideoGAN [85] Generation 3 Forerunner of video generation with GAN MocoGAN [86] Generation 3 Decomposing motion and content for video generation with GAN TemporalGAN [144] Generation 3 Decomposing temporal and image generator for video generation Video Colorizati”
Recognizing Human Actions as the Evolution of Pose Estimation Maps (2018, unknown): “70% Unsupervised Learning [29] 2017 56.”
Improving Spatiotemporal Self-Supervision by Deep Reinforcement Learning (2018, methodology): “Various surrogate tasks have been proposed, including predicting a sequence of basic motions [30], counting parts within regions [35] or embedding images into text topic spaces [39].”
Dual Motion GAN for Future-Flow Embedded Video Prediction (2017, methodology): “Various methods have been investigated to predict a future motion field [15, 33, 32, 17, 19] and visual representation [28, 30] given an image or a video sequence.”
Modality Distillation with Multiple Stream Networks for Action Recognition (2018, methodology): “The first part of the table (indicated by × symbol) refers to the unsupervised method proposed in [17], which achieve surprisingly high results even without relying on labels in learning representations.”
Graph Distillation for Action Detection with Privileged Modalities (2017, background): “NTU RGB+D is a very challenging dataset and has been recently studied in numerous studies [24,29,32,35,47].”
Learning Action Recognition Model from Depth and Skeleton Videos (2017, methodology): “We compare our performance to the state-of-the-art action recognition methods including Comparative Coding Descriptor (CCD) [5], Histogram of Oriented Gradients (HOG(2)) [14], Discriminative Virtual Views (DVV) [11], Continuous Virtual Path (CVP) [41], Histogram of Oriented 4D Normals (HON4D) [15], ”
Self-Supervised Spatiotemporal Learning via Video Clip Order Prediction (2019, unknown): “[24] 53.”
Space-Time Correspondence as a Contrastive Random Walk (2020, background): “For video, temporal information makes for natural pretext tasks, including future prediction [26, 74, 52, 47, 50], arrow of time [55, 89], motion estimation [1, 36, 78, 45] or audio [62, 2, 61, 41].”
Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction (2020, methodology): “To constrain the challenging generation of high dimensional images, several methods rather predict geometric transformations between frames [16, 24, 75] or use optical flow [46, 38, 33, 32, 31].”