Use What You Have: Video retrieval using representations from collaborative experts

By Yang Liu, Samuel Albanie, Arsha Nagrani, Andrew Zisserman. 2019

445 citations recorded by the source corpus.

The rapid growth of video on the internet has made searching for video content using natural language queries a significant challenge. Human-generated queries for video datasets `in the wild' vary a lot in terms of degree of specificity, with some queries describing specific details such as the names of famous identities, content from speech, or text available on the screen. Our goal is to condense the multi-modal, extremely high dimensional information from videos into a single, compact video representation for the task of video retrieval using free-form text queries, where the degree of specificity is open-ended. For this we exploit existing knowledge in the form of pre-trained semantic embeddings which include 'general' features such as motion, appearance, and scene features from visual content. We also explore the use of more 'specific' cues from ASR and OCR which are intermittently available for videos and find that these signals remain challenging to use effectively for retrieval. We propose a collaborative experts model to aggregate information from these different pre-trained experts and assess our approach empirically on five retrieval benchmarks: MSR-VTT, LSMDC, MSVD, DiDeMo, and ActivityNet. Code and data can be found at this http URL. This paper contains a correction to results reported in the previous version.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “The notion as well of “Mixture-of-Experts” ((Jordan and Jacobs 1994), see (Masoudnia and Ebrahimpour 2014) for a review) is a common paradigm for combining the outputs of multiple models, and specifically mixtures of experts across multimodal domains including vision and audio (Liu et al. 2019a) hav”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, methodology): “For DiDeMo and ActivityNet Captions, we follow [80, 41] to evaluate paragraph-to-video retrieval, where all sentence descriptions for a video are concatenated”
Multi-modal Transformer for Video Retrieval (2020, methodology): “describe those three components. Features F. In order to learn an eective representation from dierent modalities inherent in video data, we begin with video feature extractors called \experts" [14,16,19,31]. In contrast to previous methods, we learn a joint representation leveraging both cross-”
Support-set bottlenecks for video-text representation learning (2020, methodology): “Noteworthily, instead of using 9 and 7 diﬀerent types of visual features as in CE (Liu et al., 2019) and MMT (Gabeur et al., 2020), we use only the above 2 features and achieve on par or superior performance.”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “Some models [19, 20] also incorporate audio into the retrieval task, where the audio is considered jointly with the video (denoted by T → A+V).”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “However, existing works [41, 26, 29, 20, 50, 44] in this field are often evaluated on distinct datasets under different experimental settings, making fair comparison difficult between methods.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, background): “…outperforms most supervised DiDeMo dataset R@1 ↑ R@5 S UPERVISED S2VT (Venugopalan et al., 2014) 11.9 33.6 FSE (Zhang et al., 2018) 13.9 44.5 CE (Liu et al., 2019a) 16.1 41.1 ClipBERT (Lei et al., 2021) 20.4 48.0 Z ERO - SHOT VideoCLIP (Zero-shot) 16.6 46.9 Table 2: Text → video retrieval on…”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, methodology): “We evaluate video-paragraph retrieval following (Liu et al., 2019; Lei et al., 2021; Bain et al., 2021), where all sentence descriptions for a video are concatenated into a single query.”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “…artificial intelligence tasks such as Visual Question Answering (Antol et al. 2015; Jang et al. 2017; Gao et al. 2018; Xu et al. 2017; Fan et al. 2019) and Text-to-Video Retrieval (Liu et al. 2019; Mithun et al. 2018; Yu, Kim, and Kim 2018; Miech, Laptev, and Sivic 2018; Song and Soleymani 2019).”
Object-aware Video-language Pre-training for Retrieval (2021, methodology): “Limited by small-scale video-language datasets, previous video-language pretraining methods [12,23,26,39,41], have tended to use a combination of multiple “experts” to extract multi-modal features offline, e.”
A Straightforward Framework For Video Retrieval Using CLIP (2021, methodology): “Besides, we consistently achieve state-of-the-art results on all the recall metrics in the Full split from MSR-VTT.”
Open-book Video Captioning with Retrieve-Copy-Generate Network (2021, background): “In general, there are two kinds of model architectures, i.e ., Bi-encoders [4, 6, 19, 20] and Cross-encoders [26, 30, 33, 3].”
Video-Text Pre-training with Learned Regions (2021, methodology): “We split these videos into 1200 , 100 , and 670 as the train, val, and test set [6,41,51].”
Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning (2020, methodology): “The first type of works [5, 26, 27, 31] embeds videos and texts as global vectors that encode salient semantic meanings.”
HANet: Hierarchical Alignment Networks for Video-Text Retrieval (2021, background): “Some works [12, 25, 28] made full use of multimodal cues, e.”