Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity
By Hanwen Liang, Niamul Quader, Zhixiang Chi, Lizhe Chen, Peng Dai, Juwei Lu, Yang Wang. 2021
35 citations recorded by the source corpus.
Recent self-supervised video representation learning methods have found significant success by exploring essential properties of videos, e.g. speed, temporal order, etc. This work exploits an essential yet under-explored property of videos, the \textit{video continuity}, to obtain supervision signals for self-supervised representation learning. Specifically, we formulate three novel continuity-related pretext tasks, i.e. continuity justification, discontinuity localization, and missing section approximation, that jointly supervise a shared backbone for video representation learning. This self-supervision approach, termed as Continuity Perception Network (CPNet), solves the three tasks altogether and encourages the backbone network to learn local and long-ranged motion and context representations. It outperforms prior arts on multiple downstream tasks, such as action recognition, video retrieval, and action localization. Additionally, the video continuity can be complementary to other coarse-grained video properties for representation learning, and integrating the proposed pretext task to prior arts can yield much performance gains.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
Citations in the corpus
- Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay (2022, background): “Besides, inspired by the remarkable progress in self-supervised visual representation learning [1,2,6,7,19], [12] propose to enable continual learning without labeled data.”
- RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (2024, unknown): “Moreover, robotic data is differed significantly from images and videos with temporal and spatial continuity (Chen et al., 2019; Liang et al., 2022).”