VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding

By Hu Xu, Gargi Ghosh, Po-Yao (Bernie) Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze Luke Zettlemoyer Christoph Feichtenhofer. 2021

814 citations recorded by the source corpus.

We present VideoCLIP, a contrastive approach to pre-train a unified model for zero-shot video and text understanding, without using any labels on downstream tasks. VideoCLIP trains a transformer for video and text by contrasting temporally overlapping positive video-text pairs with hard negatives from nearest neighbor retrieval. Our experiments on a diverse series of downstream tasks, including sequence-level text-video retrieval, VideoQA, token-level action localization, and action segmentation reveal state-of-the-art performance, surpassing prior work, and in some cases even outperforming supervised approaches. Code is made available at https://github.com/pytorch/fairseq/examples/MMPT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Jianfeng Gao
3. Cordelia Schmid
4. Zhe Gan
5. Li Fei-Fei
6. Yu Qiao
7. Yejin Choi
8. Sergey Levine

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Ming-Hsuan Yang
4. Xiaogang Wang
5. Cordelia Schmid
6. Trevor Darrell
7. Andrew Zisserman
8. Jitendra Malik

In Multimodal Learning

1. Zhe Gan
2. Cordelia Schmid
3. Linjie Li
4. Lijuan Wang
5. Zicheng Liu
6. Ivan Laptev
7. Josef Sivic
8. Mohit Bansal

In Representation & Self-Supervised Learning

1. Cordelia Schmid
2. Arsha Nagrani
3. Yann LeCun
4. Andrew Zisserman
5. Zhe Gan
6. Ivan Laptev
7. Steven C. H. Hoi
8. Li Fei-Fei

Citations in the corpus

• Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “Other methods have also attempted zero-shot on MSR-VTT text-to-video retrieval (Xu et al. 2021; Miech et al. 2020; Bain et al. 2021), but these have all been outperformed by Portillo-Quintero et al. (2021).”
• MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “Recent work has explored this paradigm for matching video frames with their transcripts [121], with their audio signal [96, 114], or both [3, 2]; these works likewise perform well on single-modality tasks like audio classification and activity recognition.”
• VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “We follow [16,17] to report retrieval performance on the entire validation clips.”
• InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, background): “Pretraining large-scale video-text models and fine-tuning them for down-stream tasks has become the norm in the video-language field [21–23, 19, 24, 23, 15, 25–31, 4, 5, 32– 34].”
• TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (2023, methodology): “The first stage pre-trains the model using large-scale image/video-text pairs for vision-language alignment [17, 20, 26, 36, 38, 50].”
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “[108] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.”
• ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, methodology): “, unsupervised learning [67, 20, 76], and video-text contrastive learning [64, 79, 78, 66].”
• MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, methodology): “If φV is a monolithic model like S3D [135] in VideoCLIP [136], then the video features from every sliding window must be recomputed, which would incur a much higher cost per time step.”
• Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (2022, methodology): “We find this phenomenon consistently across various multi-modal models, covering texts, natural images [39], videos [50], medical images [53], and amino-acid sequences [11].”
• InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “Besides, the popular methods often include two or three pretraining tasks, e.g. masked language modeling [31], video-text matching [24], video-text contrastive learning [47] and video-text masked modeling [30].”
• Multimodal Large Language Models: A Survey (2023, methodology): “The CLIP model led to the development of the VideoCLIP model [57].”
• Align and Prompt: Video-and-Language Pre-training with Entity Prompts (2021, methodology): “Apart from the canonical pre-training tasks, such as masked language modeling (MLM) [10, 26, 30, 35, 48, 57] and video-text matching (VTM) [30,35], several methods [35,37,52] apply contrastive learning on offline extracted visual features.”
• R3M: A Universal Visual Representation for Robot Manipulation (2022, background): “Additionally, there are a number of works that use language to learn representations from videos [62, 63].”
• MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval (2022, background): “Methods in the first category [6, 15, 16, 18, 29, 32, 35, 42, 48, 51, 53], adopt the “dual-encoder” architectures, where two individual encoders are used to contrast global video and text representations, and contrastive learning [24, 34] is utilized to distinguish paired video-text data with…”
• VideoChat: chat-centric video understanding (2023, neutral): “Additionally, prevalent techniques often encompass two or three pretraining tasks, such as masked language modeling [22], video-text matching [43], video-text contrastive learning [49, 46], masked video modeling [41, 44, 46] and video-text masked modeling [12].”