Visual Semantic Search: Retrieving Videos via Complex Textual Queries

By Dahua Lin, Sanja Fidler, Chen Kong, Raquel Urtasun. 2014

159 citations recorded by the source corpus.

In this paper, we tackle the problem of retrieving videos using complex natural language queries. Towards this goal, we first parse the sentential descriptions into a semantic graph, which is then matched to visual concepts using a generalized bipartite matching algorithm. Our approach exploits object appearance, motion and spatial relations, and learns the importance of each term using structure prediction. We demonstrate the effectiveness of our approach on a new dataset designed for semantic search in the context of autonomous driving, which exhibits complex and highly dynamic scenes with many objects. We show that our approach is able to locate a major portion of the objects described in the query with high accuracy, and improve the relevance in video retrieval.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, background): “Lin et al. [17] does so for videos.”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
SPICE: Semantic Propositional Image Caption Evaluation (2016, methodology): “To parse an image caption into a scene graph, we use a two-stage approach similar to previous works [16,17,18].”
A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, neutral): “Visual information retrieval with natural language queries has long been tackled via joint visuallanguage embedding models [12,19,24,25,26,27,28].”
MovieQA: Understanding Stories in Movies through Question-Answering (2015, background): “In [21], the authors retrieve relevant video clips for natural language queries, while [29] exploits captioned clips to learn action and role models.”
Order-Embeddings of Images and Language (2015, methodology): “The caption-image retrieval task has become a standard evaluation of joint models of vision and language (Hodosh et al., 2013; Lin et al., 2014a).”
Localizing Moments in Video with Natural Language (2017, background): “Additionally, datasets like [20, 28] are small and restricted to spe-ciﬁc domains, such as dash-cam or cooking videos, while datasets [5, 32, 45, 50] sourced from movies and YouTube are frequently edited and tend to only include entertaining moments (see [38] for discussion).”
Grounding of Textual Phrases in Images by Reconstruction (2015, background): “In video domain some of the representative works on spatial-temporal language grounding are [22] and [39].”
Learning Language-Visual Embedding for Movie Understanding with Natural-Language (2016, methodology): “Image-Caption Retrieval: Another alternative is image-caption retrieval that has been deﬁned as a standard way to evaluate joint language-visual models [8,11,12,24].”
Monocular Object Instance Segmentation and Depth Ordering with CNNs (2015, background): “It is also important for, e.g., image captioning, Q&A and retrieval techniques [18, 26, 24] where describing a 3D scene is easier and potentially more informative than describing a soup of orderless object detections [18].”
TALL: Temporal Activity Localization via Language Query (2017, methodology): “[14] parsed the sentence descriptions into a semantic graph, which are then matched to visual concepts in the videos using a generalized bipartite matching algorithm.”
Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval (2015, methodology): “Lin et al. (2014a) also use rules to transform dependency trees into semantic graphs which they use for video search.”
Instance-Level Segmentation for Autonomous Driving with Deep Densely Connected MRFs (2015, background): “An instance mask is also more informative than pixel-wise class labeling as it allows counting, important for applications such as retrieval [21].”
Evaluation of Text Generation: A Survey (2020, methodology): “A two-stage process is typically used to parse an image caption into a scene graph (Schuster et al., 2015; Lin et al., 2014).”
Question Relevance in VQA: Identifying Non-Visual And False-Premise Questions (2016, methodology): “We use NeuralTalk2 (Karpathy and Fei-Fei, 2015) pre-trained on the MSCOCO dataset (Lin et al., 2014b) (images and associated captions) to generate a caption C for the given image, and then compute a learned similarity between Q and C (details below).”
VSE++: Improved Visual-Semantic Embeddings (2017, methodology): “Older work (Lin et al. (2014a)) performed matching between words and objects based on classiﬁcation scores.”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2017, methodology): “Lin et al. [29] retrieve videos from a sentence description using an intermediate graph representation.”
Visual Coreference Resolution in Visual Dialog using Neural Module Networks (2018, background): “Most works in this area focus on the specific task of localizing a textual referential expression in the image [21,25,32,36,39,45,50] or video [38,27,49,7].”
Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech (2018, background): “Some have studied correspondence matching between categorical abstractions, such as words and objects [20, 21, 22, 23, 24, 25].”