Correlation Hashing Network for Efficient Cross-Modal Retrieval

By Mingsheng Long, Jianmin Wang. 2016

63 citations recorded by the source corpus.

Hashing is widely applied to approximate nearest neighbor search for large-scale multimodal retrieval with storage and computation efficiency. Cross-modal hashing improves the quality of hash coding by exploiting semantic correlations across different modalities. Existing cross-modal hashing methods first transform data into low-dimensional feature vectors, and then generate binary codes by another separate quantization step. However, suboptimal hash codes may be generated since the quantization error is not explicitly minimized and the feature representation is not jointly optimized with the binary codes. This paper presents a Correlation Hashing Network (CHN) approach to cross-modal hashing, which jointly learns good data representation tailored to hash coding and formally controls the quantization error. The proposed CHN is a hybrid deep architecture that constitutes a convolutional neural network for learning good image representations, a multilayer perception for learning good text representations, two hashing layers for generating compact binary codes, and a structured max-margin loss that integrates all things together to enable learning similarity-preserving and high-quality hash codes. Extensive empirical study shows that CHN yields state of the art cross-modal retrieval performance on standard benchmarks.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Giovanni Maria Farinella
2. Antonino Furnari
3. Kristen Grauman
4. Dima Damen
5. Ling Shao
6. Jitendra Malik
7. Luc Van Gool
8. Dacheng Tao

In Information Retrieval & Search

1. Liang Wang
2. Evangelos Kazakos
3. Ling Shao
4. Fumin Shen
5. Yuming Shen
6. Michael Wray
7. Dima Damen
8. Jiawei Zhan

In Multimodal Learning

1. Ling Shao
2. Li Liu
3. Shu Wu
4. Qiyue Yin
5. Giovanni Maria Farinella
6. Antonino Furnari
7. Liang Wang
8. Kaiye Wang

Citations in the corpus

• Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “mAP has been used in other retrieval works [98,100,101,102], yet it only considers relevance between items to be binary.”
• A Comprehensive Survey on Cross-modal Retrieval (2016, methodology): “…[63], CMFH [64], LSSH [65] Nonlinear modeling MSAE [66], DMHOR [67] Pairwise based methods Linear modeling CMSSH [68], CRH [69], IMVH [70], QCH[71] RaHH [72], HTH [73] Nonlinear modeling MLBE [74], PLMH [75], MM-NN [76], CHN [77] Supervised methods Linear modeling SM 2 H [78], DCDH [79], SCM [80]”
• CVM-Net: Cross-View Matching Network for Image-Based Ground-to-Aerial Geo-Localization (2018, background): “The most widely used loss function in image retrieval task is the max-margin triplet loss [9, 3, 8, 11, 24, 42, 47, 14, 32, 19] that enforces the distances of positive pairs to be less than the distances of negative pairs.”
• Joint-modal Distribution-based Similarity Hashing for Large-scale Unsupervised Deep Cross-modal Retrieval (2020, methodology): “For the supervised meth-ods [2, 3, 9, 15, 18, 19, 40, 42, 45], manually-annotated labels, requiring expensive human cost and computation, are leveraged to construct the similarity matrices and supervise hash code generation.”
• Deep Sketch Hashing: Fast Free-Hand Sketch-Based Image Retrieval (2017, methodology): “In fact, among various hashing methods, cross-modality hashing [31, 68, 72, 26, 32, 2, 57, 75, 71, 10, 23, 5, 4], which learns binary codes by preserving the correlations between heterogeneous representations from different modalities, are more related to SBIR problems.”