CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment

By Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Rui Song, Houqiang Li, Jiebo Luo. 2022

115 citations recorded by the source corpus.

The pre-trained image-text models, like CLIP, have demonstrated the strong power of vision-language representation learned from a large scale of web-collected image-text data. In light of the well-learned visual features, some existing works transfer image representation to video domain and achieve good results. However, how to utilize image-language pre-trained model (e.g., CLIP) for video-language pre-training (post-pretraining) is still under explored. In this paper, we investigate two questions: 1) what are the factors hindering post-pretraining CLIP to further improve the performance on video-language tasks? and 2) how to mitigate the impact of these factors? Through a series of comparative experiments and analyses, we find that the data scale and domain gap between language sources have great impacts. Motivated by these, we propose a Omnisource Cross-modal Learning method equipped with a Video Proxy mechanism on the basis of CLIP, namely CLIP-ViP. Extensive results show that our approach improves the performance of CLIP on video-text retrieval by a large margin. Our model also achieves SOTA results on a variety of datasets, including MSR-VTT, DiDeMo, LSMDC, and ActivityNet. We will release our code and pre-trained CLIP-ViP models at https://github.com/microsoft/XPretrain/tree/main/CLIP-ViP.

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

All in One: Exploring Unified Video-Language Pre-Training (2022, neutral): “4 - - CLIP-ViP [54] T+V+CE WIT300M+HD-VILA-100M - 225.”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, background): “The main reason is that image encoders are compatible with arbitrary frame sampling strategies and enable a more flexible frame-to-video feature aggregation scheme, as observed in Xue et al. (2023).”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, background): “They have also been extended to other modalities, such as audio [8, 19] and video [20].”