Heterogeneous Metric Learning with Joint Graph Regularization for Cross-Media Retrieval

By Xiaohua Zhai, Yuxin Peng, Jianguo Xiao. 2013

128 citations recorded by the source corpus.

As the major component of big data, unstructured heterogeneous multimedia content such as text, image, audio, video and 3D increasing rapidly on the Internet. User demand a new type of cross-media retrieval where user can search results across various media by submitting query of any media. Since the query and the retrieved results can be of different media, how to learn a heterogeneous metric is the key challenge. Most existing metric learning algorithms only focus on a single media where all of the media objects share the same data representation. In this paper, we propose a joint graph regularized heterogeneous metric learning (JGRHML) algorithm, which integrates the structure of different media into a joint graph regularization. In JGRHML, different media are complementary to each other and optimizing them simultaneously can make the solution smoother for both media and further improve the accuracy of the final metric. Based on the heterogeneous metric, we further learn a high-level semantic metric through label propagation. JGRHML is effective to explore the semantic relationship hidden across different modalities. The experimental results on two datasets with up to five media types show the effectiveness of our proposed approach.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Liang Wang
3. Yi Yang
4. Xuelong Li
5. Yuxin Peng
6. Heng Tao Shen
7. Cheng Deng
8. Tien-Ping Tan

In Representation & Self-Supervised Learning

1. Xiangteng He
2. Liang Wang
3. Kaiye Wang
4. Xin Huang
5. Yunzhen Zhao
6. Xinbo Gao
7. Ran He
8. Dacheng Tao

Citations in the corpus

• Fine-Grained Image Classification via Combining Vision and Language (2017, methodology): “Zhai et al. [24] propose to learn projection functions by the metric learning, and this method is further improved as Joint Representation Learning (JRL) [ 25] by adding other information such as semantic categories and semi-supervised information.”
• Joint Feature Selection and Subspace Learning for Cross-Modal Retrieval (2016, methodology): “The model defines a Markov random field on the document level which allows modeling more flexible document similarities.”
• Learning Cross-Modal Common Representations by Private–Shared Subspaces Separation (2020, background): “3) NUS-WIDE-10k [56] is a subset cropped from the original NUS-WIDE [57] dataset containing around 270 000 image–text pairs from 81 classes.”
• Cross-Modal Similarity Learning: A Low Rank Bilinear Formulation (2014, methodology): “The proposed formulation is introduced and derived to find the solution detailed in the next part.”
• A Comprehensive Survey on Cross-modal Retrieval (2016, methodology): “Multi-NPP [30], MVML-GL [31], JGRHML [32] Deep learning methods RGDBN [33], MSDS [34] Rank based methods Shallow methods SSI [35], PAMIR [36], LSCMR [37], bi-LSCMR [38], Wsabie [39], RCCA [40] Deep learning methods DeViSE [41], DT-RNNs [42], Deep Fragment [43], C 2 MLR [44] Supervised methods…”
• Triplet-Based Deep Hashing Network for Cross-Modal Retrieval (2018, background): “graph regularization [6] [7], and dictionary learning [8], are based on subspace learning [9]–[11], which maps different modality data into a common subspace and measures the similarities in the common space.”
• Deep Supervised Cross-Modal Retrieval (2019, methodology): “The representative methods of this subclass include the Multiview Metric Learning with Global consistency and Local smoothness (MVML-GL) method [38], the Joint Graph Regularised Heterogeneous Metric Learning (JGRHML) method [39] and the Modality-Specific Deep Structure (MSDS) method[31].”
• An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges (2017, methodology): “For example, the work of [7] can be classified as both a metric learning and graph regularization method.”