CLIP2Video: Mastering Video-Text Retrieval via Image CLIP

By Han Fang, Pengfei Xiong, Luhui Xu, Yu Chen. 2021

366 citations recorded by the source corpus.

We present CLIP2Video network to transfer the image-language pre-training model to video-text retrieval in an end-to-end manner. Leading approaches in the domain of video-and-language learning try to distill the spatio-temporal video features and multi-modal interaction between videos and languages from a large-scale video-text dataset. Different from them, we leverage pretrained image-language model, simplify it as a two-stage framework with co-learning of image-text and enhancing temporal relations between video frames and video-text respectively, make it able to train on comparatively small datasets. Specifically, based on the spatial semantics captured by Contrastive Language-Image Pretraining (CLIP) model, our model involves a Temporal Difference Block to capture motions at fine temporal video frames, and a Temporal Alignment Block to re-align the tokens of video clips and phrases and enhance the multi-modal correlation. We conduct thorough ablation studies, and achieve state-of-the-art performance on major text-to-video and video-to-text retrieval benchmarks, including new records of retrieval accuracy on MSR-VTT, MSVD and VATEX.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, methodology): “Multiple methods utilize the CLIP model in a straightforward way to perform text-to-video retrieval (Fang et al., 2021; Luo et al., 2021; Narasimhan et al., 2021).”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, neutral): “Finetuning JEMC (Mithun et al. 2018) 12.5 32.1 42.4 16.0 yes Collaborative Experts (Liu et al. 2019a) 20.6 50.3 64.0 5.3 15.6 40.9 55.2 8.3 yes SSB (Patrick et al. 2020) 28.5 58.6 71.6 3.0 no CLIP4Clip (Luo et al. 2021) 43.1 70.5 81.2 2.0 no ViT-V/32 CLIP2Video (Fang et al. 2021) 43.5 72.3 82.1 2.0 ”
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, background): “Numerous studies have proved its efficacy, involving 2D images, videos, texts, and audio [15, 17, 48], and enhance the cross-modal performance for downstream tasks [5, 25, 37, 61], and video-text-audio integration for text generation [36].”
Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model (2022, background): “, CLIP [18] and ALIGN [10], emerges and finds its applications in few-shot or zero-shot learning tasks [5, 28].”
Disentangled Representation Learning for Text-Video Retrieval (2022, background): “Researchers [34,2,29,11,13] explore the power of this design principle to text-video retrieval with additional temporal fusion module.”
CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation (2021, methodology): “Recently, pre-trained CLIP has been used for several zero-shot downstream applications [16, 18, 26, 41,45,54].”
MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, methodology): “CLIP is even adopted in temporal domains [Guzhov et al. 2021; Luo et al. 2021; Fang et al. 2021] that utilize large datasets of video sequences that are paired with text and audio.”
ActionCLIP: A New Paradigm for Video Action Recognition (2021, background): “Vision-text multi-modality is a hot topic in several vision-text related fields recently, like pre-training [20, 21], vision-text retrieval [10, 31] and so on.”
CLIP2TV: An Empirical Study on Transformer-based Methods for Video-Text Retrieval (2021, background): “Recent works like [11] and [5] have showed the beneﬁts of image-text retrieval pre-training and advantages of end-to-end training for video-text retrieval task.”
VLP: A Survey on Vision-language Pre-training (2022, methodology): “CLIP4Clip[17] Video ViT-PFs Emb Dual-Stream No VLC WebVid2M+CC3M VLR”
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, methodology): “…some works directly adapt image-text pre-trained models to video-text downstream tasks without further pre-training on video data (Luo et al., 2021; Fang et al., 2021; Gorti et al., 2022; Zhao et al., 2022), while still outperforming models pre-trained on video data (Xu et al., 2021b; Bain et al.,…”
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, neutral): “…in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks (Li et al., 2020; Akbari et al., 2021; Fang et al., 2021; Yan et al., 2021; Li et al., 2021; Radford et al., 2021; Li et al., 2022; Zellers et al., 2022; Zeng et al., 2022b; Yang et al.,…”
Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss (2021, background): “…retrieval (Faghri et al. 2017; Frome et al. 2013; Gong et al. 2014; Gu et al. 2018), though developed for years, still largely follows the single-stream or two-stream architecture (Dzabraev et al. 2021; Zhu and Yang 2020; Lei et al. 2021; Gabeur et al. 2020; Fang et al. 2021; Luo et al. 2021).”
CRIS: CLIP-Driven Referring Image Segmentation (2021, background): “Recently, the remarkable success of the CLIP [39] has shown its capability of learning SOTA image-level visual concepts from 400 million image-text pairs, which assists many multi-modal tasks achieve significant improvements, including image-text retrieval [39], video-text retrieval [6, 31].”
ImageBind One Embedding Space to Bind Them All (2023, methodology): “Various methods adapt CLIP to extract semantically strong video representations [15, 43, 45, 79].”