Video Representation Learning with Visual Tempo Consistency
By Ceyuan Yang, Yinghao Xu, Bo Dai, Bolei Zhou. 2020
94 citations recorded by the source corpus.
Visual tempo, which describes how fast an action goes, has shown its potential in supervised action recognition. In this work, we demonstrate that visual tempo can also serve as a self-supervision signal for video representation learning. We propose to maximize the mutual information between representations of slow and fast videos via hierarchical contrastive learning (VTHCL). Specifically, by sampling the same instance at slow and fast frame rates respectively, we can obtain slow and fast video frames which share the same semantics but contain different visual tempos. Video representations learned from VTHCL achieve the competitive performances under the self-supervision evaluation protocol for action recognition on UCF-101 (82.1\%) and HMDB-51 (49.2\%). Moreover, comprehensive experiments suggest that the learned representations are generalized well to other downstream tasks including action detection on AVA and action anticipation on Epic-Kitchen. Finally, we propose Instance Correspondence Map (ICM) to visualize the shared semantics captured by contrastive learning.
Subfields: Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Cordelia Schmid
- Jitendra Malik
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Andrew Zisserman
Citations in the corpus
- Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
- Spatiotemporal Contrastive Video Representation Learning (2020, methodology): “Examples abound, including models for predicting the future [36, 27, 18], changing temporal sampling rates [46], and sorting video frames or clips [26, 24, 45].”
- Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity (2021, methodology): “Most contrastive learning-based methods build upon the instance discrimination objective and encourage the model to learn some temporal invariance of video instances (Han, Xie, and Zisserman 2019, 2020a; Wang, Jiao, and Liu 2020; Han, Xie, and Zisserman 2020b; Tao, Wang, and Yamasaki; Yang et al. 20”
- A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (2021, methodology): “Unsupervised learning in videos has followed a similar trajectory with earlier methods focusing on predictive tasks based on motion, color and spatiotemporal ordering [29,43,1, 44,79,86,61,85,59,58,21,51,87,67,22,48,93,16,88,71,45], and contrastive objectives with visual [75, 80, 34, 53, 28, 94] and”
- Anticipative Video Transformer (2021, background): “Researchers explore a variety of “free” supervisory signals, such as temporal consistency [21, 40, 43, 91, 95], inter-frame predictability [35, 36, 39, 80], and cross-modal correspon-”