CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval

By Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wenli Lei, Nan Duan, Tianrui Li. 2021

1,161 citations recorded by the source corpus.

Video-text retrieval plays an essential role in multi-modal research and has been widely used in many real-world web applications. The CLIP (Contrastive Language-Image Pre-training), an image-language pre-training model, has demonstrated the power of visual concepts learning from web collected image-text datasets. In this paper, we propose a CLIP4Clip model to transfer the knowledge of the CLIP model to video-language retrieval in an end-to-end manner. Several questions are investigated via empirical studies: 1) Whether image feature is enough for video-text retrieval? 2) How a post-pretraining on a large-scale video-text dataset based on the CLIP affect the performance? 3) What is the practical mechanism to model temporal dependency between video frames? And 4) The Hyper-parameters sensitivity of the model on video-text retrieval task. Extensive experimental results present that the CLIP4Clip model transferred from the CLIP can achieve SOTA results on various video-text retrieval datasets, including MSR-VTT, MSVC, LSMDC, ActivityNet, and DiDeMo. We release our code at https://github.com/ArrowLuo/CLIP4Clip.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Multimodal Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, neutral): “Finetuning JEMC (Mithun et al. 2018) 12.5 32.1 42.4 16.0 yes Collaborative Experts (Liu et al. 2019a) 20.6 50.3 64.0 5.3 15.6 40.9 55.2 8.3 yes SSB (Patrick et al. 2020) 28.5 58.6 71.6 3.0 no CLIP4Clip (Luo et al. 2021) 43.1 70.5 81.2 2.0 no ViT-V/32 CLIP2Video (Fang et al. 2021) 43.5 72.3 82.1 2.0 ”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, background): “Future work may explore the uniﬁed per-Task Text Retrieval Video Retrieval MSVD MSVD R@1 R@5 R10 R@1 R@5 R@10 CLIP4clip [48] 56”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “2− 2% on R@1 over endto-end pre-trained models, excluding CLIP [76, 78].”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “Recent work [31, 44, 9] show that with image-text pre-training, models trained on 2D features alone can achieve decent performance on many video-and-language (VidL) tasks.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, methodology): “There are concurrent works (Luo et al., 2021; Portillo-Quintero et al., 2021) about using im-”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, methodology): “Besides, pretraining UMT [39] with our Panda-5M also outperforms the existing state-of-the-art methods [49, 50, 74] which are pretrained with much more vision-text data pairs ( i.e ., > 100M).”
X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval (2022, methodology): “Different from the previous work [38] that only consider the coarse-grained contrast, our proposed XCLIP adopt multi-grained contrast during retrieval.”
CLIP2TV: An Empirical Study on Transformer-based Methods for Video-Text Retrieval (2021, methodology): “In CLIP4Clip [11], CLIP [15] encoders are adopted in both sides.”
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, methodology): “For SeqTransformer, we follow the seqTransf type in CLIP4Clip (Luo et al., 2021) and the residual connection in their implementation.”
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval (2022, methodology): “We also pre-train a model initialized from CLIP weights following the setting of [31] on CC3M and WebVid-2M.”
LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, neutral): “While it’s generally expected that using more frames boosts performance, previous studies (Luo et al., 2021; Lei et al., 2021; 2022) have noted that performance tends to plateau when training with more than 16 frames.”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “Figure 1: InternVideo delivers the best performance on extensive video-related tasks, compared with the state-of-the-art methods (including specialized [1–5] and foundation models [6–9]).”
Align and Prompt: Video-and-Language Pre-training with Entity Prompts (2021, methodology): “Note that we do not compare with the work [35] which uses the powerful encoders from CLIP [43] pretrained on 400M Method PT datasets R1↑ R5↑ R10↑ MdR↓”
Open-Vocabulary Panoptic Segmentation with MaskCLIP (2022, methodology): “Many algorithms have been developed lately [34, 31, 36, 24, 26, 28] trying knowledge distillation from the CLIP model to beneﬁt the down-stream tasks one way or the other by leveraging the rich semantic language information paired in the images.”
MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, methodology): “CLIP is even adopted in temporal domains [Guzhov et al. 2021; Luo et al. 2021; Fang et al. 2021] that utilize large datasets of video sequences that are paired with text and audio.”
CLIP2Video: Mastering Video-Text Retrieval via Image CLIP (2021, methodology): “Meanwhile, adopting CLIP for ﬁne-tuning and modeling interaction between frames with temporal transformer, the performance can be further increased, which is shown by CLIP4clip [23] on Tab.3.”
CRIS: CLIP-Driven Referring Image Segmentation (2021, background): “Recently, the remarkable success of the CLIP [39] has shown its capability of learning SOTA image-level visual concepts from 400 million image-text pairs, which assists many multi-modal tasks achieve significant improvements, including image-text retrieval [39], video-text retrieval [6, 31].”