Dual Encoding for Zero-Example Video Retrieval

By Jianfeng Dong, Xirong Li, Chaoxi Xu, S. Ji, Yuan He, Gang Yang, Xun Wang. 2018

299 citations recorded by the source corpus.

This paper attacks the challenging problem of zero-example video retrieval. In such a retrieval paradigm, an end user searches for unlabeled videos by ad-hoc queries described in natural language text with no visual example provided. Given videos as sequences of frames and queries as sequences of words, an effective sequence-to-sequence cross-modal matching is required. The majority of existing methods are concept based, extracting relevant concepts from queries and videos and accordingly establishing associations between the two modalities. In contrast, this paper takes a concept-free approach, proposing a dual deep encoding network that encodes videos and queries into powerful dense representations of their own. Dual encoding is conceptually simple, practically effective and end-to-end. As experiments on three benchmarks, i.e. MSR-VTT, TRECVID 2016 and 2017 Ad-hoc Video Search show, the proposed solution establishes a new state-of-the-art for zero-example video retrieval.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, background): “Similarly to [19], we enable our network to ingest both dynamic video and still images.”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “We compare the NCE based approach (ours) to the frequently used max margin ranking loss [19, 21, 33, 40, 51, 55, 79, 80, 83, 84] and a binary classification loss (i.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “A common approach to model vision and language is learning a joint embedding space where visual and textual cues are adjacent if and only if they are semantically similar [6, 8, 10, 11, 25, 32, 35, 37, 38, 59, 54, 55, 57].”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “We follow the same evaluation protocol for video-to-text retrieval as used in prior work (Liu et al., 2019a; Dong et al., 2019; 2018; Mithun et al., 2018), which reports the minimum rank among all valid text captions for a given video query, and each test video is associated with 20 captions.”
Support-set bottlenecks for video-text representation learning (2020, background): “, 1998) were shown to effectively capture temporal dependencies in videos (Dong et al., 2019), we integrate CNNs into our transformer pooling heads to better capture video temporal signals.”
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment (2021, background): “Some works have relied on attention mechanisms to extract key information from videos [45, 55], while others preserve visual information by composing pairwise joint representation using 3D tensors [53] or use multi-level video encoders to separately encode the spatial and temporal cues [11].”
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021, methodology): “As can be observed, our model with MMP generalizes well on YouTube2Text, outperforming HGR (Chen et al., 2020a) by 7.6 and DualEncoder (Dong et al., 2019) by 10.3 in R@1.”
COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning (2020, background): “Modalities such as audio, text and motion provide strong cues to learn richer spatio-temporal features [67, 68, 16, 17, 9, 69, 11].”
X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval (2022, methodology): “Recent years have witnessed the rapid development of VTR, which is supported by a series of pre-training multi-modal models [4, 30, 44], innovative retrieval methods [3, 5, 13–15, 24, 30, 34, 35, 38, 41, 54, 58, 61, 63, 66] and video-text benchmarks [2, 6, 7, 45, 56].”
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers (2021, methodology): “Other non-Transformer based vision and text approaches used recurrent neural networks [14, 15, 36], MLP [71, 72], or bagof-words [19, 51] text models.”
HANet: Hierarchical Alignment Networks for Video-Text Retrieval (2021, unknown): “In this case, most existing works [8, 11, 22, 29] focus on how to learn global representations of videos and texts, and achieve this goal by leverage various embedding networks, e.”
Action Modifiers: Learning From Adverbs in Instructional Videos (2019, background): “Several problems are at the intersection between language and video: captioning [24, 38, 55, 59], retrieval [9, 16, 21, 31, 33, 52, 54] and visual question answering [15, 56, 57, 61].”
CLIP2Video: Mastering Video-Text Retrieval via Image CLIP (2021, methodology): “In the past years, remarkable progress [40, 43, 18, 21, 13, 11, 27, 41, 20, 9, 6, 3] has occurred across many video-text benchmarks [15, 39, 38, 7, 32, 16].”
Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning (2020, methodology): “To address the limitation of keyword-based approach, more and more researchers are paying attention to video retrieval using natural language texts that contain richer and more structured details than keywords, a.k.a, cross-modal video-text retrieval [5, 27, 43].”
Syntax-Aware Action Targeting for Video Captioning (2020, unknown): “It has extensive applications such as Visual Question Answering (VQA) [57, 16], human-robot interaction [32] and video retrieval [64, 14].”