Predicting Visual Features From Text for Image and Video Caption Retrieval

By Jianfeng Dong, Xirong Li, Cees G. M. Snoek. 2017

243 citations recorded by the source corpus.

This paper strives to find amidst a set of sentences the one best describing the content of a given image or video. Different from existing works, which rely on a joint subspace for their image and video caption retrieval, we propose to do so in a visual space exclusively. Apart from this conceptual novelty, we contribute Word2VisualVec , a deep neural network architecture that learns to predict a visual feature representation from textual input. Example captions are encoded into a textual embedding based on multiscale sentence vectorization and further transferred into a deep visual feature of choice via a simple multilayer perceptron. We further generalize Word2VisualVec for video caption retrieval, by predicting from text both three-dimensional convolutional neural network features as well as a visual-audio representation. Experiments on Flickr8k, Flickr30k, the Microsoft Video Description dataset, and the very recent NIST TrecVid challenge for video caption retrieval detail Word2VisualVec's properties, its benefit over textual embeddings, the potential for multimodal query composition, and its state-of-the-art results.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “We follow the same evaluation protocol for video-to-text retrieval as used in prior work [55, 90, 91, 70], which reports the minimum rank among all valid text captions for a given video query, and each test video is associated with 20 captions.”
Dual Encoding for Zero-Example Video Retrieval (2018, methodology): “We initialize the matrix using a word2vec [23] model provided by [7], which trained word2vec on English tags of 30 million Flickr images.”
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings (2019, background): “This approach has a number of advantages over training a single embedding space as is standardly done [7, 8, 15, 22, 24].”
COCO-CN for Cross-Lingual Image Tagging, Captioning, and Retrieval (2018, methodology): “We enhance W2VV [28] by 1) adding an attention layer after GRU to take into account all the hidden vectors, and 2) substituting the original mean square error for the contrastive loss.”
Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning (2020, unknown): “SE [21] 5.0 16.4 24.6 47 215.1 7.7 20.3 31.2 28 185.8 105.2 VSE++ [6] 5.7 17.1 24.8 65 300.8 10.2 25.4 35.1 25 228.1 118.3 Mithum et al. [27] 5.8 17.6 25.2 61 296.6 10.5 26.7 35.9 25 266.6 121.7 W2VV [4] 6.1 18.7 27.5 45 - 11.8 28.9 39.1 21 - 132.1 Dual Encoding [5] 7.7 22.0 31.8 32 - 13.0 30.8 43.3”
Synthesized Feature based Few-Shot Class-Incremental Learning on a Mixture of Subspaces (2021, background): “Learning without forgetting using word vectors: Word vectors have shown promising success on various computer vision tasks such as zero-shot learning, few-shot learning, image/video captioning and visual question answering [15, 29, 39, 43, 9, 46, 47, 6, 5, 7].”