A Straightforward Framework For Video Retrieval Using CLIP

By Jes'us Andr'es Portillo-Quintero, José Carlos Ortiz-Bayliss, Hugo Terashima-Mar'in. 2021

143 citations recorded by the source corpus.

Video Retrieval is a challenging task where a text query is matched to a video or vice versa. Most of the existing approaches for addressing such a problem rely on annotations made by the users. Although simple, this approach is not always feasible in practice. In this work, we explore the application of the language-image model, CLIP, to obtain video representations without the need for said annotations. This model was explicitly trained to learn a common space where images and text can be compared. Using various techniques described in this document, we extended its application to videos, obtaining state-of-the-art results on the MSR-VTT and MSVD benchmarks.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Multimodal Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “Since our demonstrated system uses exactly the method of Portillo-Quintero et al. (2021) for its processing of CLIP features but additionally incorporates LLM reasoning on speech-to-text transcripts, the increased measured performance of our method (i.e. 27.2→ 42.8 R@1 on MSR-VTT 1k-A) directly…”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, methodology): “There are concurrent works (Luo et al., 2021; Portillo-Quintero et al., 2021) about using im-”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, methodology): “Like our work, the concurrent work from Portillo-Quintero et al. (2021) is also built on the CLIP for video-text retrieval.”
Disentangled Representation Learning for Text-Video Retrieval (2022, methodology): “In recent studies [26,35,34,29], the text and video inputs are usually considered separately for efficient deployment through a Bi-Encoder architecture [38,18].”
Simple but Effective: CLIP Embeddings for Embodied AI (2021, background): “CLIP has also proven to be effective at video tasks particularly text based video retrieval [9, 17, 21].”
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (2023, methodology): “Furthermore, we achieved state-of-the-art performance in video-to-text retrieval tasks, significantly outperforming CLIP-straight.”
BridgeFormer: Bridging Video-text Retrieval with Multiple Choice Questions (2022, methodology): “text representations, which is pre-trained on 400 million image-text pairs, some recent work [24, 30] utilize the pretrained CLIP for text-to-video retrieval.”
CLIP2TV: An Empirical Study on Transformer-based Methods for Video-Text Retrieval (2021, methodology): “Comparing with them, even the CLIP under zero-shot setting [14] surpass the most of the feature-level methods, indicating the beneﬁts and necessity of large scale image-text pre-training for video-text retrieval.”
CLIP2Video: Mastering Video-Text Retrieval via Image CLIP (2021, methodology): “We achieve the SOTA results on all three datasets compared with baselines, where the visible growth of performance can be found by employing CLIP [29] as pre-trained model.”