Self-supervised Co-training for Video Representation Learning

By Tengda Han, Weidi Xie, Andrew Zisserman. 2020

398 citations recorded by the source corpus.

The objective of this paper is visual-only self-supervised video representation learning. We make the following contributions: (i) we investigate the benefit of adding semantic-class positives to instance-based Info Noise Contrastive Estimation (InfoNCE) training, showing that this form of supervised contrastive learning leads to a clear improvement in performance; (ii) we propose a novel self-supervised co-training scheme to improve the popular infoNCE loss, exploiting the complementary information from different views, RGB streams and optical flow, of the same data source by using one view to obtain positive class samples for the other; (iii) we thoroughly evaluate the quality of the learnt representation on two different downstream tasks: action recognition and video retrieval. In both cases, the proposed approach demonstrates state-of-the-art or comparable performance with other self-supervised approaches, whilst being significantly more efficient to train, i.e. requiring far less training data to achieve similar performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

Citations in the corpus

• Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, neutral): “Examples include the motion and appearance of a video [30, 80], depth and appearance [36, 91], luminance and chrominance of an image [80, 91], or as in our work sound and video [2, 4, 12, 63].”
• Dense Contrastive Learning for Self-Supervised Visual Pre-Training (2020, neutral): “Generally speaking, the success of self-supervised learning [41, 17, 42, 47, 16, 14] can be attributed to two important aspects namely contrastive learning, and pretext tasks.”
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (2021, neutral): “(V+A+T), with the latter training on a Audioset (AS) [23] and HowTo100M (HT) [60], and CoCLR [35] can be seen as a variant of MoCo on rgb and optical-flow input.”
• Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation (2021, neutral): “For the COCLR [11] method, the positive samples of a video clip in RGB space are determined by the closeness of clips in the optical flow space and vice versa.”
• Dissecting Supervised Constrastive Learning (2021, neutral): “Recent works (Khosla et al., 2020; Han et al., 2020) have shown that this is indeed possible via a supervised variant of a contrastive loss (Chopra et al., 2005; Hadsell et al., 2006) that has full access to label information.”
• Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing (2023, neutral): “, video) to capture temporal information [30, 55, 78, 80].”
• With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations (2021, neutral): “Related to our work, [27] uses nearest neighbor retrieval to define selfsupervision for video representation learning across different modalities (e.”
• Self-Supervised Representation Learning: Introduction, advances, and challenges (2021, neutral): “State-of-the-art procedures successfully leverage large source data sets and approaches but do not yet outperform supervised pretraining [78].”
• Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, neutral): “To study the learned representations in isolation we take the state-action representations φ(s, a) trained on the image-based point NineRooms task, and run a linear probe [3, 51] experiment to see whether the representations have learned to encode task-relevant information (the shortest path distanc”
• Broaden Your Views for Self-Supervised Video Learning (2021, neutral): “In this direction also, video contrastive methods have been very successful [18, 31, 32, 67].”
• Long-Short Temporal Contrastive Learning of Video Transformers (2021, neutral): “In comparison, recent approaches in selfsupervised learning leverage contrastive learning [9, 11, 12, 14, 28, 29, 51].”