Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition

By Deepti Ghadiyaram, Matt Feiszli, Du Tran, Xueting Yan, Heng Wang, Dhruv Kumar Mahajan. 2019

319 citations recorded by the source corpus.

Current fully-supervised video datasets consist of only a few hundred thousand videos and fewer than a thousand domain-specific labels. This hinders the progress towards advanced video architectures. This paper presents an in-depth study of using large volumes of web videos for pre-training video models for the task of action recognition. Our primary empirical finding is that pre-training at a very large scale (over 65 million videos), despite on noisy social-media videos and hashtags, substantially improves the state-of-the-art on three challenging public action recognition datasets. Further, we examine three questions in the construction of weakly-supervised video action datasets. First, given that actions involve interactions with objects, how should one construct a verb-object pre-training label space to benefit transfer learning the most? Second, frame-based models perform quite well on action recognition; is pre-training for good image features sufficient or is pre-training for spatio-temporal features valuable for optimal transfer learning? Finally, actions are generally less well-localized in long videos vs. short videos; since action labels are provided at a video level, how should one choose video clips for best performance, given some fixed budget of number or minutes of videos?

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, methodology): “The analysis of R(2+1)D BERT architecture is limited to RGB modality, since the study [9] of the IG65M dataset where R(2+1)D architecture is preferred is limited to RGB modality.”
Learning Video Representations from Textual Web Supervision (2020, methodology): “Other approaches use user-deﬁned keywords or tags [20, 28, 43, 19] or algorithmically-generated topics [1, 31] to the same end.”
Self-Supervised MultiModal Versatile Networks (2020, background): “Dataset abbreviations: AudioSet, HowTo100M, Instagram65M [26], SoundNet [10], 2M videos from YouTube8M [5], Kinetics600; their length in years is given in the “years” column.”
Multiview Transformers for Video Recognition (2022, methodology): “In the second part of the tables, we compare to methods that are pre-trained on web-scale datasets such as Instagram 65M [25], JFT-300M [61], JFT-3B [80], WTS [60], Florence [79] or HowTo100M [47].”
Support-set bottlenecks for video-text representation learning (2020, methodology): “Large-scale training data has enabled the more eﬀective pretraining of image (Sun et al., 2017; Yalniz et al., 2019), video (Ghadiyaram et al., 2019; Thomee et al., 2016) and textual representations (Raﬀel et al., 2019).”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “A variety of tasks, focused primarily on text-video modelling, have been applied to these datasets including: task segmentation [19–21], reference resolution [22, 23], action segmentation [5], video clip ordering [24–27], and action recognition [28–30].”
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning (2021, unknown): “6Some SotA models are pre-trained with extremely large (weakly-)supervised datasets, e.g., IG65M [24] in [34] and JFT-300M [58] in [2].”
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval (2022, unknown): “D model pre-trained on IG65M [17]).”
Video Understanding as Machine Translation (2020, methodology): “The video feature extractor is implemented as a R(2+1)D-34 network [49] pre-trained on the IG65M dataset [11].”
Self-Supervised Pretraining of 3D Features on any Point-Cloud (2021, background): “Pretraining visual features on large labeled datasets is a pre-requisite to achieve good performance when access to annotations is limited [27, 48, 55, 90].”