Scalable Deep Multimodal Learning for Cross-Modal Retrieval
By Peng Hu, Liangli Zhen, Dezhong Peng, Pei Liu. 2019
129 citations recorded by the source corpus.
Cross-modal retrieval takes one type of data as the query to retrieve relevant data of another type. Most of existing cross-modal retrieval approaches were proposed to learn a common subspace in a joint manner, where the data from all modalities have to be involved during the whole training process. For these approaches, the optimal parameters of different modality-specific transformations are dependent on each other and the whole model has to be retrained when handling samples from new modalities. In this paper, we present a novel cross-modal retrieval method, called Scalable Deep Multimodal Learning (SDML). It proposes to predefine a common subspace, in which the between-class variation is maximized while the within-class variation is minimized. Then, it trains m modality-specific networks for m modalities (one network for each modality) to transform the multimodal data into the predefined common subspace to achieve multimodal learning. Unlike many of the existing methods, our method can train different modality-specific networks independently and thus be scalable to the number of modalities. To the best of our knowledge, the proposed SDML could be one of the first works to independently project data of an unfixed number of modalities into a predefined common subspace. Comprehensive experimental results on four widely-used benchmark datasets demonstrate that the proposed method is effective and efficient in multimodal learning and outperforms the state-of-the-art methods in cross-modal retrieval.
Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
In Information Retrieval & Search
In Multimodal Learning
Citations in the corpus
- Dynamic Modality Interaction Modeling for Image-Text Retrieval (2021, background): “It benefits a variety of applications, ranging from cross-modal retrieval [11, 12, 23, 41] tomultimedia recommendation [36].”