Deep Visual-Semantic Hashing for Cross-Modal Retrieval

By Yue Cao, Mingsheng Long, Jianmin Wang, Qiang Yang, Philip S. Yu. 2016

259 citations recorded by the source corpus.

Due to the storage and retrieval efficiency, hashing has been widely applied to approximate nearest neighbor search for large-scale multimedia retrieval. Cross-modal hashing, which enables efficient retrieval of images in response to text queries or vice versa, has received increasing attention recently. Most existing work on cross-modal hashing does not capture the spatial dependency of images and temporal dynamics of text sentences for learning powerful feature representations and cross-modal embeddings that mitigate the heterogeneity of different modalities. This paper presents a new Deep Visual-Semantic Hashing (DVSH) model that generates compact hash codes of images and sentences in an end-to-end deep learning architecture, which capture the intrinsic cross-modal correspondences between visual data and natural language. DVSH is a hybrid deep architecture that constitutes a visual-semantic fusion network for learning joint embedding space of images and text sentences, and two modality-specific hashing networks for learning hash functions to generate compact binary codes. Our architecture effectively unifies joint multimodal embedding and cross-modal hashing, which is based on a novel combination of Convolutional Neural Networks over images, Recurrent Neural Networks over sentences, and a structured max-margin objective that integrates all things together to enable learning of similarity-preserving and high-quality hash codes. Extensive empirical evidence shows that our DVSH approach yields state of the art results in cross-modal retrieval experiments on image-sentences datasets, i.e. standard IAPR TC-12 and large-scale Microsoft COCO.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Multimodal Learning

Citations in the corpus

Achieving Human Parity on Visual Question Answering (2021, background): “Current approaches on cross-modal retrieval generally falls into two categories: (1) real-valued representation learning [32, 61, 92], and (2) binary representation learning [7, 12, 47], which is also called cross-modal hashing.”
Deep Hashing Network for Unsupervised Domain Adaptation (2017, background): “The closest work incorporating hashing and adaptation appears in cross-modal hashing, where deep hashing techniques embed multi-modal data and learn hash codes for two related domains, like text and images [5, 6, 29].”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, background): “Multimodal representation learning frameworks can be divided into three types: (a) joint representation which aims to learn a shared semantic subspace [63,71,26,3,48,51,95]; (b) an encoder-decoder framework which aims to translate from one modality into another and keep their semantics consistent [3”
Robust Unsupervised Cross-modal Hashing for Multimedia Retrieval (2020, methodology): “Supervised CMH methods leverage available semantic affinities [2, 3, 14] constructed by labels or label information of training data [4, 5, 9, 12, 16, 17, 43, 48] to supervise the learning process.”
Correlation Hashing Network for Efficient Cross-Modal Retrieval (2016, background): “However, our method shares the same problem setting with DCMH that only requires similarity labels across images and texts, while DVSH further requires bimodal image-text pairs to learn modalshared representation.”
Deep Sketch Hashing: Fast Free-Hand Sketch-Based Image Retrieval (2017, methodology): “In fact, among various hashing methods, cross-modality hashing [31, 68, 72, 26, 32, 2, 57, 75, 71, 10, 23, 5, 4], which learns binary codes by preserving the correlations between heterogeneous representations from different modalities, are more related to SBIR problems.”
SCH-GAN: Semi-Supervised Cross-Modal Hashing by Generative Adversarial Network (2018, background): “Representative deep learning-based methods include deep and bidirectional representation learning model [19], deep visual-semantic hashing (DVH) [22], and cross-media neural network hashing (CMNNH) [21].”
Self-Supervised Adversarial Hashing Networks for Cross-Modal Retrieval (2018, background): “Recently, some works have reported on deep crossmodal hashing retrieval [3, 9, 12, 37].”
A Decade Survey of Content Based Image Retrieval Using Deep Learning (2020, background): “Semantic hashing has been also extensively performed for sketch based image retrieval [180], [167], [209], [146], cross-modal retrieval [192], [205], [183].”
Finding Similar Exercises in Online Education Systems (2018, background): “Multimodal learning is a powerful approach to deal with multiple heterogeneous data, such as sound and video [24], video and text [42], or image and text [3, 6].”
Deep Supervised Cross-Modal Retrieval (2019, background): “A variety of approaches have been proposed to learn such a common representation space, which can be roughly divided into two categories: 1) binary-valued representation learning [9, 3, 41], also called as cross-modal hashing, and 2) real-valued representation leaning [30, 19, 21].”
Multimodal Machine Learning: A Survey and Taxonomy (2017, background): “Coordinated Similarity Images + Text [61], [105] Video + Text [159], [231] Structured Images + Text [32], [212], [248] Audio + Articulatory [220]”
Fine-Grained Similarity Measurement between Educational Videos and Exercises (2020, background): “What’s more, many multimodal learning methods on handling images (frames) and text can also be applied for modeling segments [3, 7, 17, 21, 27].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Initially, most of the work was concentrated on combining low-level linguistic units such as words with images or videos for building visual-semantic embeddings (Frome et al., 2013; Kiros et al., 2014b; Liu et al., 2015; Cao et al., 2016; Tsai et al., 2017; Guo et al., 2018; Wang et al., 2019) usefu”
Triplet-Based Deep Hashing Network for Cross-Modal Retrieval (2018, methodology): “In the development of deep neural networks (DNN), deep models are applied to crossmodal hashing [17] [24]–[26] to efficiently capture the correlations between heterogeneous data.”
Two-Stream Deep Hashing With Class-Specific Centers for Supervised Image Search (2020, background): “Recent deep learning-based hashing methods [28], [44], which combine feature learning and hash code quantization, have shown that deep networks can contribute to more effective hash code learning.”
Zero-Shot Sketch-Image Hashing (2018, methodology): “As an extension of conventional data hashing techniques [51, 20, 39, 54, 53], cross-modal hashing [4, 13, 72, 36, 33, 26, 5, 6] show great potential in retrieving heterogeneous data with high efficiency due to the computationally cheap Hamming space matching, which is recently adopted to large-scale”
Deep Cross-Modal Hashing (2016, methodology): “Six state-of-the-art cross-modal hashing methods are adopted as baselines for comparison, including SePH [22], STMH [33], SCM [35], CMFH [7], CCA [11] and DVSH [3].”
Pairwise Relationship Guided Deep Hashing for Cross-Modal Retrieval (2017, methodology): “Deep Visual-Semantic Hashing (DVSH) (Cao et al. 2016) utilizes Convolutional Neural Networks (CNNs) and Long ShortTerm Memory (LSTM) to separately learn unified binary codes for each modality.”
Shared Predictive Cross-Modal Deep Quantization (2018, unknown): “Unlike these existing deep cross-modal methods [17], [18],”