Joint-modal Distribution-based Similarity Hashing for Large-scale Unsupervised Deep Cross-modal Retrieval

By Song Liu, Shengsheng Qian, Yang Guan, Jiawei Zhan, Long Ying. 2020

218 citations recorded by the source corpus.

Hashing-based cross-modal search which aims to map multiple modality features into binary codes has attracted increasingly attention due to its storage and search efficiency especially in large-scale database retrieval. Recent unsupervised deep cross-modal hashing methods have shown promising results. However, existing approaches typically suffer from two limitations: (1) They usually learn cross-modal similarity information separately or in a redundant fusion manner, which may fail to capture semantic correlations among instances from different modalities sufficiently and effectively. (2) They seldom consider the sampling and weighting schemes for unsupervised cross-modal hashing, resulting in the lack of satisfactory discriminative ability in hash codes. To overcome these limitations, we propose a novel unsupervised deep cross-modal hashing method called Joint-modal Distribution-based Similarity Hashing (JDSH) for large-scale cross-modal retrieval. Firstly, we propose a novel cross-modal joint-training method by constructing a joint-modal similarity matrix to fully preserve the cross-modal semantic correlations among instances. Secondly, we propose a sampling and weighting scheme termed the Distribution-based Similarity Decision and Weighting (DSDW) method for unsupervised cross-modal hashing, which is able to generate more discriminative hash codes by pushing semantic similar instance pairs closer and pulling semantic dissimilar instance pairs apart. The experimental results demonstrate the superiority of JDSH compared with several unsupervised cross-modal hashing methods on two public datasets NUS-WIDE and MIRFlickr.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Chenliang Li
2. Ming Yan
3. Fei Huang
4. Wei Wang
5. Bin Bi
6. Songfang Huang
7. Ji Zhang
8. Haiyang Xu

In Information Retrieval & Search

1. Ming Yan
2. Shengsheng Qian
3. Songfang Huang
4. Wei Wang

In Multimodal Learning

1. Fei Huang
2. Chenliang Li
3. Haoqi Fan
4. Haiyang Xu
5. Ming Yan
6. Shengsheng Qian
7. Ji Zhang
8. Weihua Chen

Citations in the corpus

• Achieving Human Parity on Visual Question Answering (2021, background): “Visual question answering involves the cross-modal fusion and alignment between image and text, which is related to the topic of cross-modal retrieval [12, 26, 28, 32, 47, 60, 61, 84, 92].”
• HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval (2021, background): “Cross-modal Retrieval [58, 10, 13, 67, 8, 3, 35, 9, 11, 46, 59, 60, 25, 57, 29, 42] has attracted the increasing attention with the aim to search the semantic similar samples from different modalities.”