X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval

By Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, Rongrong Ji. ACM Multimedia, 2022

500 citations recorded by the source corpus.

Video-text retrieval has been a crucial and fundamental task in multi-modal research. The development of video-text retrieval has been considerably promoted by large-scale multi-modal contrastive pre-training, which primarily focuses on coarse-grained or fine-grained contrast. However, cross-grained contrast, which is the contrast between coarse-grained representations and fine-grained representations, has rarely been explored in prior research. Compared with fine-grained or coarse-grained contrasts, cross-grained contrast calculate the correlation between coarse-grained features and each fine-grained feature, and is able to filter out the unnecessary fine-grained features guided by the coarse-grained feature during similarity calculation, thus improving the accuracy of retrieval. To this end, this paper presents a novel multi-grained contrastive model, namely X-CLIP, for video-text retrieval. However, another challenge lies in the similarity aggregation problem, which aims to aggregate fine-grained and cross-grained similarity matrices to instance-level similarity. To address this challenge, we propose the Attention Over Similarity Matrix (AOSM) module to make the model focus on the contrast between essential frames and words, thus lowering the impact of unnecessary frames and words on retrieval results. With multi-grained contrast and the proposed AOSM module, X-CLIP achieves outstanding performance on five widely-used video-text retrieval datasets, including MSR-VTT (49.3 R@1), MSVD (50.4 R@1), LSMDC (26.1 R@1), DiDeMo (47.8 R@1) and ActivityNet (46.2 R@1).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1ec886e2235763b08fa606a5d5ea3f4540f715ec

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training (2022, methodology): “First, several methods directly transfer image-text models to video-text tasks by simply concatenating video frame [22,24] or building an extra temporal encoder [31,32].”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, background): “Its various temporal variants yield competitive or state-of-the-art performance in several core tasks [5, 22].”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, methodology): “Besides, pretraining UMT [39] with our Panda-5M also outperforms the existing state-of-the-art methods [49, 50, 74] which are pretrained with much more vision-text data pairs ( i.e ., > 100M).”