Geometry Guided Convolutional Neural Networks for Self-Supervised Video Representation Learning

By Chuang Gan, Boqing Gong, Kun Liu, Hao Su, Leonidas J. Guibas. 2018

126 citations recorded by the source corpus.

It is often laborious and costly to manually annotate videos for training high-quality video recognition models, so there has been some work and interest in exploring alternative, cheap, and yet often noisy and indirect training signals for learning the video representations. However, these signals are still coarse, supplying supervision at the whole video frame level, and subtle, sometimes enforcing the learning agent to solve problems that are even hard for humans. In this paper, we instead explore geometry, a grand new type of auxiliary supervision for the self-supervised learning of video representations. In particular, we extract pixel-wise geometry information as flow fields and disparity maps from synthetic imagery and real 3D movies, respectively. Although the geometry and high-level semantics are seemingly distant topics, surprisingly, we find that the convolutional neural networks pre-trained by the geometry cues can be effectively adapted to semantic video understanding tasks. In addition, we also find that a progressive training strategy can foster a better neural network for the video recognition task than blindly pooling the distinct sources of geometry cues together. Extensive results on video dynamic scene recognition and action recognition tasks show that our geometry guided networks significantly outperform the competing methods that are trained with other types of labeling-free supervision signals.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, background): “It was shown that KD has several benefits for optimization and knowledge transfer between tasks [59, 13, 14].”
Contrastive Multiview Coding (2019, neutral): “ouring frame Method # of Views UCF-101 HMDB-51 Random - 48.2 19.5 ImageNet - 67.7 28.0 VGAN* [57] 2 52.1 - LT-Motion* [35] 2 53.0 - TempCoh [41] 1 45.4 15.9 Shufﬂe and Learn [39] 1 50.2 18.1 Geometry [17] 2 55.1 23.3 OPN [33] 1 56.3 22.1 ST Order [8] 1 58.6 25.0 Cross and Learn [51] 2 58.7 27.2 CMC ”
A critical analysis of self-supervision, or what we can learn from a single image (2019, methodology): “…et al., 2015; Gao et al., 2016; Wang et al., 2017), foreground-background segmentation via video segmentation (Pathak et al., 2017), optical-ﬂow (Gan et al., 2018; Mahendran et al., 2018), future-frame synthesis (Srivastava et al., 2015), audio prediction from video (de Sa, 1994; Owens et al.,…”
Self-labelling via simultaneous clustering and representation learning (2019, methodology): “, 2017), optical-flow (Gan et al., 2018; Mahendran et al., 2018), futureframe synthesis (Srivastava et al.”
Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. (2018, neutral): “2DCNN GeometryGuided [11] 54.”
Cross Pixel Optical Flow Similarity for Self-Supervised Learning (2018, neutral): “This was used as a pretraining strategy by [7] to learn representations for pairs of frames .”
Unsupervised Learning From Video With Deep Neural Embeddings (2019, methodology): “3D-CNNs with dense equal sampling.”
Spatiotemporal Contrastive Video Representation Learning (2020, background): “Furthermore, videos can often supply multimodal signals for cross-modality self-supervision, such as geometric cues [19], speech or language [60, 59, 45], audio [40, 3, 51, 4], optical flow [30] or combinations of multiple modalities [2] and tasks [52].”
A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (2021, methodology): “Unsupervised learning in videos has followed a similar trajectory with earlier methods focusing on predictive tasks based on motion, color and spatiotemporal ordering [29,43,1, 44,78,85,60,84,58,57,21,51,86,66,22,48,91,16,87,70,45], and contrastive objectives with visual [74, 79, 34, 53, 28, 92] and”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “[293] explored geometry as a new type of practical auxiliary knowledge for self-supervised learning of video representations.”
Curious Representation Learning for Embodied Intelligence (2021, background): “Approaches towards unsupervised learning include colorizing images [50], predicting image rotations [18] and geometry transformation [15], solving jigsaw puzzles [27], and adversarial inference [13].”
Graph Convolutional Networks for Temporal Action Localization (2019, background): “Understanding human actions in videos has been becoming a prominent research topic in computer vision, owing to its various applications in security surveillance, human behavior analysis and many other areas [10, 35, 38, 12, 13, 14, 15, 16, 42].”
RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning (2020, unknown): “How to learn effective video representations with a few annotations even without annotations is an important yet challenging task [13, 11, 10].”
Discriminability Distillation in Group Representation Learning (2020, methodology): “Frame-based methods [64,14,46,19] firstly extract frame features and aggregate them.”