Online multimodal deep similarity learning with application to image retrieval

By Pengcheng Wu, Steven C. H. Hoi, Hao Xia, Peilin Zhao, Dayong Wang, Chunyan Miao. 2013

173 citations recorded by the source corpus.

Recent years have witnessed extensive studies on distance metric learning (DML) for improving similarity search in multimedia information retrieval tasks. Despite their successes, most existing DML methods suffer from two critical limitations: (i) they typically attempt to learn a linear distance function on the input feature space, in which the assumption of linearity limits their capacity of measuring the similarity on complex patterns in real-world applications; (ii) they are often designed for learning distance metrics on uni-modal data, which may not effectively handle the similarity measures for multimedia objects with multimodal representations. To address these limitations, in this paper, we propose a novel framework of online multimodal deep similarity learning (OMDSL), which aims to optimally integrate multiple deep neural networks pretrained with stacked denoising autoencoder. In particular, the proposed framework explores a unified two-stage online learning scheme that consists of (i) learning a flexible nonlinear transformation function for each individual modality, and (ii) learning to find the optimal combination of multiple diverse modalities simultaneously in a coherent process. We conduct an extensive set of experiments to evaluate the performance of the proposed algorithms for multimodal image retrieval tasks, in which the encouraging results validate the effectiveness of the proposed technique.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, positive): “The training procedure in AEs involves minimizing reconstruction error, i.e., the output and input showing minimal difference.”
Learning Fine-Grained Image Similarity with Deep Ranking (2014, background): “[25] employs deep learning architecture to learn ranking model, but it learns deep network from the “hand-crafted features” rather than directly from the pixels.”
Convolutional Neural Network for Paraphrase Identification (2015, neutral): “We set β = 2 (cf. Wu et al. (2013)).”
Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics (2014, background): “Other examples that apply multi-modal deep learning use restricted Boltzmann machines (Sri-vastava and Salakhutdinov, 2012; Feng et al., 2013), auto-encoders (Wu et al., 2013) or recursive neural networks (Socher et al., 2014).”
Deep Learning for Content-Based Image Retrieval: A Comprehensive Study (2014, methodology): “The entire CNN model can be updated by back-propagation algorithm following the same scheme in [52].”
MultiGranCNN: An Architecture for General Matching of Text Chunks on Multiple Levels of Granularity (2015, methodology): “This model computes the match score of two g-phrases as their similarity using a radial basis function kernel: where we set β = 2 (cf. Wu et al. (2013)).”
Gated Multimodal Units for Information Fusion (2017, methodology): “…using neural network architectures (Andrew et al., 2013; Feng et al., 2013; Kang et al., 2012; Kiros et al., 2014a; Lu et al., 2014; Mao et al., 2014; Tu et al., 2014; Wu et al., 2013) with two input layers separately and including a ﬁnal supervised layer such as softmax regression classiﬁer.”
Learning Grounded Meaning Representations with Autoencoders (2014, background): “…focused on projecting words and images into a common space using a variety of deep learning methods ranging from deep and restricted Boltzman machines (Srivastava and Salakhutdinov, 2012; Feng et al., 2013), to au-toencoders (Wu et al., 2013), and recursive neural networks (Socher et al., 2013b).”
DeepStore: In-Storage Acceleration for Intelligent Queries (2019, background): “, the keys in transactional database and key-value store), and can comprise of images/image-patches [92, 96, 106], sketch [70], color map [91], context map [98, 99], music [72, 97], and many more, intelligent query systems also leverage DNNs to automatically extract the features from query input to ”
Comparative Deep Learning of Hybrid Representations for Image Recommendations (2016, methodology): “Motivated by previous efforts on deep ranking models [14, 27, 30], we propose a comparative learning method to tackle the imperfect training data.”
A Decade Survey of Content Based Image Retrieval Using Deep Learning (2020, background): “retrieval task [66].”
Learning Concept Embeddings with Combined Human-Machine Expertise (2015, background): “Finally, an interesting branch of work revolves around teaching CNNs to satisfy triplet questions as part of the overall pipeline [35, 38].”
Localizing and Orienting Street Views Using Overhead Imagery (2016, background): “The fourth network that we call the triplet network or ranking network, shown in Figure 3(c), is popular for image feature learning and retrieval [35, 31, 24, 32, 26, 25], though its eﬀectiveness has not been explored in cross-view image matching.”
Deep Multimodal Learning: A Survey on Recent Advances and Trends (2017, methodology): “Multimodal deep learning has been used in problems involving tissue and organ segmentation [17], multimodal medical image retrieval [18], multimodal medical image registration [19], and computer-aided diagnosis [20].”
A Deep Structured Model with Radius–Margin Bound for 3D Human Activity Recognition (2015, methodology): “Our model is inspired by the effectiveness of two widely successful techniques: deep learning [21, 14, 20, 16, 48, 26, 45] and the latent structured models [56, 11, 1, 32, 24].”
Online Learning: A Comprehensive Survey (2018, background): “…related works: Other than solving classification tasks, online multi-view learning has been explored for solving similarity learning or distance metric learning, such as Online multimodal deep similarity learning (Wu et al., 2013) and online multi-modal distance metric learning (Wu et al., 2016).”