Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles

By Dahun Kim, In-So Kweon. 2018

369 citations recorded by the source corpus.

Self-supervised tasks such as colorization, inpainting and zigsaw puzzle have been utilized for visual representation learning for still images, when the number of labeled images is limited or absent at all. Recently, this worthwhile stream of study extends to video domain where the cost of human labeling is even more expensive. However, the most of existing methods are still based on 2D CNN architectures that can not directly capture spatio-temporal information for video applications. In this paper, we introduce a new self-supervised task called as Space-Time Cubic Puzzles to train 3D CNNs using large scale video dataset. This task requires a network to arrange permuted 3D spatio-temporal crops. By completing Space-Time Cubic Puzzles, the network learns both spatial appearance and temporal relation of video frames, which is our final goal. In experiments, we demonstrate that our learned 3D representation is well transferred to action recognition tasks, and outperforms state-of-the-art 2D CNN-based competitors on UCF101 and HMDB51 datasets.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “For video data, these pretext tasks learn representations by ordering video frames [1, 18, 34, 41, 51, 79, 83], tracking [62, 77], or using cross-modal signals like audio [2, 3, 19, 36, 60, 61].”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Examples include predicting the future frame [86], motion and appearance statistics [95], speed [9, 96] and encodings [62, 42, 43], sorting frames or video clips [58, 103, 49, 35].”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, methodology): “In this section, we compare our 3DRotNet with other self-supervised methods on action recognition task including the 2DCNN-based [13, 29, 30, 47, 49] and the 3DCNN-based methods [43, 22].”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, neutral): “7 3D-ST-Puzzle [33] 16×112 63.”
Unsupervised Learning From Video With Deep Neural Embeddings (2019, methodology): “We initialize networks by pretrained weights on Kinetics and then finetune them on these datasets, following the procedures used for the most recent work [16, 32, 18].”
Compositional Few-Shot Recognition with Primitive Discovery and Enhancing (2020, methodology): “, the upper and lower parts of dogs and cars can be easily distinguished, although they are not highly semantically related), self-supervision from object structures [31, 42] may help to alleviate the influence of semantic gaps.”
A Survey on Contrastive Self-supervised Learning (2020, neutral): “4† ImageNet-inflated [63] R3D 60.”
A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (2021, methodology): “Unsupervised learning in videos has followed a similar trajectory with earlier methods focusing on predictive tasks based on motion, color and spatiotemporal ordering [29,43,1, 44,79,86,61,85,59,58,21,51,87,67,22,48,93,16,88,71,45], and contrastive objectives with visual [75, 80, 34, 53, 28, 94] and”
Self-Supervised Learning by Cross-Modal Audio-Video Clustering (2019, background): “Some works have extended image pretext tasks to video [28, 72, 79].”
Parameter Efficient Multimodal Transformers for Video Representation Learning (2020, neutral): “ST-Puzzle (Kim et al., 2019), ClipOrder (Xu et al.”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, background): “These include predicting geometric transformations [24], jigsaw puzzles [26] and performing noise contrastive retrieval [17].”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, methodology): “Mehtod SubCategory Code Contribution VideoGAN [85] Generation 3 Forerunner of video generation with GAN MocoGAN [86] Generation 3 Decomposing motion and content for video generation with GAN TemporalGAN [144] Generation 3 Decomposing temporal and image generator for video generation Video Colorizati”
Self-Supervised Spatio-Temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics (2019, methodology): “[20] proposed to learn spatio-temporal representations with unlabeled video data, by solving space-time cubic puzzles, which is a straightforward extension of the 2D puzzles [29].”
RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning (2020, neutral): “Unless otherwise stated, we sample 16 consecutive frames with 112 × 112 spatial size for each clip following Kim et al. [28].”