Learning Spatiotemporal Features via Video and Text Pair Discrimination

By Tianhao Li, Limin Wang. 2020

64 citations recorded by the source corpus.

Current video representations heavily rely on learning from manually annotated video datasets which are time-consuming and expensive to acquire. We observe videos are naturally accompanied by abundant text information such as YouTube titles and Instagram captions. In this paper, we leverage this visual-textual connection to learn spatiotemporal features in an efficient weakly-supervised manner. We present a general cross-modal pair discrimination (CPD) framework to capture this correlation between a video and its associated text. Specifically, we adopt noise-contrastive estimation to tackle the computational issue imposed by the huge amount of pair instance classes and design a practical curriculum learning strategy. We train our CPD models on both standard video dataset (Kinetics-210k) and uncurated web video dataset (Instagram-300k) to demonstrate its effectiveness. Without further fine-tuning, the learnt models obtain competitive results for action classification on Kinetics under the linear classification protocol. Moreover, our visual model provides an effective initialization to fine-tune on downstream tasks, which yields a remarkable performance gain for action recognition on UCF101 and HMDB51, compared with the existing state-of-the-art self-supervised training methods. In addition, our CPD model yields a new state of the art for zero-shot action recognition on UCF101 by directly utilizing the learnt visual-textual embeddings. The code will be made available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, background): “Another recent approach, like ours, uses text data which is paired with videos, such as titles [40].”
What makes for good views for contrastive learning (2020, background): “In practice, v1 and v2 are two views of the data x, such as different augmentations of the same image [58, 4, 23, 8, 7], different image channels [53], or video and text pairs [52, 36, 33].”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “Starting from the development of image-text pretraining, large-scale video-text pretraining with speciﬁc downstream task ﬁnetuning has become the standard paradigm in the video-language area [26, 30, 32, 46– 51].”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “The common paradigm in the field of video foundation models is now characterized by the combination of extensive large-scale video-language pretraining, followed by fine-tuning on specific downstream tasks [45,46,48,57,77,89,96,111].”
CLIP2Video: Mastering Video-Text Retrieval via Image CLIP (2021, background): “Others [18, 14, 35, 19] collect videos and accompanied text information from YouTube and Instagram to to learn spatio-temporal features in an efﬁcient weakly-supervised manner.”
A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (2021, methodology): “Unsupervised learning in videos has followed a similar trajectory with earlier methods focusing on predictive tasks based on motion, color and spatiotemporal ordering [29,43,1, 44,79,86,61,85,59,58,21,51,87,67,22,48,93,16,88,71,45], and contrastive objectives with visual [75, 80, 34, 53, 28, 94] and”
VideoChat: chat-centric video understanding (2023, unknown): “Early methods [38, 60] employed pretrained visual and language encoders to derive offline video and text features; however, more recent approaches [23, 27, 13, 11, 41, 44] have demonstrated the effectiveness of end-to-end training.”
Video Understanding as Machine Translation (2020, background): “The most informative such source is associated metadata, such as tags or titles from social media, which have shown strong transfer performance [11, 27].”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, methodology): “…videos with masked inputs [He et al., 2022, Tong et al., 2022, Wang et al., 2023b, Feichtenhofer et al., 2022], aligning videos with languages [Li and Wang, 2020, Xu et al., 2021, Yan et al., 2022, Li et al., 2023e], and predicting the next token using videos [Alayrac et al., 2022, Sun et…”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “The noise-contrastive loss function, which has recently been shown to yield good performance for images and videos [52, 19, 63, 41, 8, 35, 17, 56, 33], circumvents the need to implicitly specify what information needs to be discarded via a designed task.”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, methodology): “Early techniques [28, 29] used pretrained visual and language encoders to obtain ofﬂine video and text features, but recent methods [22, 21, 15, 25, 35, 36] highlight the advantages of end-to-end training.”
Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, background): “Examples include different frames of video [69, 112, 81, 37, 31, 38], point clouds [100], multiple sensory data [65, 18, 74], text and its context [63, 101, 59, 51], or video and language [84, 62, 56].”