DistInit: Learning Video Representations Without a Single Labeled Video

By Rohit Girdhar, Du Tran, Lorenzo Torresani, Deva Ramanan. 2019

60 citations recorded by the source corpus.

Video recognition models have progressed significantly over the past few years, evolving from shallow classifiers trained on hand-crafted features to deep spatiotemporal networks. However, labeled video data required to train such models has not been able to keep up with the ever increasing depth and sophistication of these networks. In this work we propose an alternative approach to learning video representations that requires no semantically labeled videos, and instead leverages the years of effort in collecting and labeling large and clean still-image datasets. We do so by using state-of-the-art models pre-trained on image datasets as “teachers” to train video models in a distillation framework. We demonstrate that our method learns truly spatiotemporal features, despite being trained only using supervision from still-image networks. Moreover, it learns good representations across different input modalities, using completely uncurated raw video data sources and with different 2D teacher models. Our method obtains strong transfer performance, outperforming standard techniques for bootstrapping video architectures with image based models by 16%. We believe that our approach opens up new approaches for learning spatiotemporal representations from unlabeled video data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, background): “Historically, this was the usual route [28] as labels were more readily available for images, e.”
Spatiotemporal Contrastive Video Representation Learning (2020, background): “[23] propose to learn video representations by distillation from image representations.”
Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition (2020, background): “Tran et al. (Tran et al. 2019) and Qiu et al. (Qiu, Yao, and Mei 2017) propose to decompose the 3D convolution into 2D spatial convolution and 1D temporal convolution.”
Speech2Action: Cross-Modal Supervision for Action Recognition (2020, methodology): “DisInit (RGB) [13] R(2+1)D-18 [41] Kinetics∗∗ 54.”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “In the context of self-supervised learning, prior work assigns weak image labels to video instances as supervision for video-level models [25].”
Omni-sourced Webly-supervised Learning for Video Recognition (2020, background): “Besides, to take advantage of representation learned from large-scale images for videos, we have to take extra steps to transfer the 2D ConvNets to the 3D counterparts, either by inflating [3] or distillation [14], and then perform fine-tuning on the target dataset, which is tedious and may be subop”