Robust Unsupervised Cross-modal Hashing for Multimedia Retrieval

By Miaomiao Cheng, Liping Jing, Michael Kwok-Po Ng. 2020

48 citations recorded by the source corpus.

With the quick development of social websites, there are more opportunities to have different media types (such as text, image, video, etc.) describing the same topic from large-scale heterogeneous data sources. To efficiently identify the inter-media correlations for multimedia retrieval, unsupervised cross-modal hashing (UCMH) has gained increased interest due to the significant reduction in computation and storage. However, most UCMH methods assume that the data from different modalities are well paired. As a result, existing UCMH methods may not achieve satisfactory performance when partially paired data are given only. In this article, we propose a new-type of UCMH method called robust unsupervised cross-modal hashing (RUCMH). The major contribution lies in jointly learning modal-specific hash function, exploring the correlations among modalities with partial or even without any pairwise correspondence, and preserving the information of original features as much as possible. The learning process can be modeled via a joint minimization problem, and the corresponding optimization algorithm is presented. A series of experiments is conducted on four real-world datasets (Wiki, MIRFlickr, NUS-WIDE, and MS-COCO). The results demonstrate that RUCMH can significantly outperform the state-of-the-art unsupervised cross-modal hashing methods, especially for the partially paired case, which validates the effectiveness of RUCMH.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Chenliang Li
2. Songfang Huang
3. Bin Bi
4. Ming Yan
5. Fei Huang
6. Wei Wang
7. Haiyang Xu
8. Luo Si

In Information Retrieval & Search

1. Ming Yan
2. Zhen-Duo Chen
3. Xin Luo
4. Songfang Huang
5. Wei Wang
6. Xin-Shun Xu
7. Yongxin Wang

In Multimodal Learning

1. Fei Huang
2. Chenliang Li
3. Haiyang Xu
4. Ming Yan
5. Ji Zhang
6. Weihua Chen
7. Fan Wang
8. Songfang Huang

Citations in the corpus

• Achieving Human Parity on Visual Question Answering (2021, background): “Visual question answering involves the cross-modal fusion and alignment between image and text, which is related to the topic of cross-modal retrieval [12, 26, 28, 32, 47, 60, 61, 84, 92].”
• High-Dimensional Sparse Cross-Modal Hashing with Fine-Grained Similarity Embedding (2021, methodology): “Representative methods include Inter-Media Hashing (IMH) [43], Collective Matrix Factorization Hashing (CMFH) [9], Composite Correlation Quantization (CCQ) [32], Fusion Similarity Hashing (FSH) [29], Robust and Flexible Discrete Hashing (RFDH) [47], Collective Reconstructive Embedding (CRE) [16], Ro”