Cross-modal Retrieval with Correspondence Autoencoder

By Fangxiang Feng, Xiaojie Wang, Ruifan Li. 2014

585 citations recorded by the source corpus.

The problem of cross-modal retrieval, e.g., using a text query to search for images and vice-versa, is considered in this paper. A novel model involving correspondence autoencoder (Corr-AE) is proposed here for solving this problem. The model is constructed by correlating hidden representations of two uni-modal autoencoders. A novel optimal objective, which minimizes a linear combination of representation learning errors for each modality and correlation learning error between hidden representations of two modalities, is used to train the model as a whole. Minimization of correlation learning error forces the model to learn hidden representations with only common information in different modalities, while minimization of representation learning error makes hidden representations are good enough to reconstruct input of each modality. A parameter $\alpha$ is used to balance the representation learning error and the correlation learning error. Based on two different multi-modal autoencoders, Corr-AE is extended to other two correspondence models, here we called Corr-Cross-AE and Corr-Full-AE. The proposed models are evaluated on three publicly available data sets from real scenes. We demonstrate that the three correspondence autoencoders perform significantly better than three canonical correlation analysis based models and two popular multi-modal deep models on cross-modal retrieval tasks.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

A Retrieve-and-Edit Framework for Predicting Structured Outputs (2018, background): “A related idea is cross-modal embeddings, which jointly embed and align items in different domains (such as images and captions) using autoencoders [39, 2, 31, 10].”
Text-to-image Synthesis via Symmetrical Distillation Networks (2018, unknown): “As a multimedia technology, cross-modal retrieval [6, 20, 29, 38, 43] has become a highlighted topic of research in the big data era.”
HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval (2021, background): “Several excellent works [66, 8, 45, 57, 11, 14, 8, 3, 36, 10, 12] are introduced to address this task.”
A Survey of Multi-View Representation Learning (2016, unknown): “[19] propose a correspondence autoencoder (Corr-AE) via constructing correlations between hidden representations of two uni-modal deep autoencoders.”
A Decade Survey of Content Based Image Retrieval Using Deep Learning (2020, methodology): “(2014) have utilized the correspondence autoencoder (Corr-AE) for cross-modal retrieval [151].”
Deep multi-view learning methods: A review (2021, background): “Recently, due to the powerful feature abstraction ability, deep learning methods [31] have vast inroads into many applications with outstanding performance, such as computer vision [20, 32, 33, 34] and artificial intelligence [19, 35, 36, 22].”
Deep Supervised Cross-Modal Retrieval (2019, methodology): “This category includes unsupervised approaches [2, 5, 33], pairwise approaches [38, 39, 31] and supervised approaches [32, 28].”
Deep Compositional Cross-modal Learning to Rank via Local-Global Alignment (2015, methodology): “Rather than a linear projection via Pearson coeﬃcient, lots of methods like DeViSE [7] and Correspondence Autoencoder [6] have been proposed to use a nonlinear projection of the images and text[18, 12, 5].”
An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges (2017, background): “A large part of the aforementioned methods are non-convolutional and take hand-crafted features as inputs as [12], [36].”
Deep Visual-Semantic Hashing for Cross-Modal Retrieval (2016, methodology): “Typical learning criteria include reconstruction error minimization [8, 37], similarity preservation as graph-based hashing [22, 33], and quantization error minimization as correlation quantization [39, 29].”
Cross-modal deep metric learning with multi-task regularization (2017, methodology): “For fair comparison, the dataset partition and feature extraction is strictly the same with [13] and [10].”
Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-Modal Pretraining (2021, background): “Cross-modal retrieval [43, 12, 48, 4, 44, 8] emerges as a promising avenue for efficient indexing and searching among large-scale data with different modalities, and is widely used in search engines [2, 14], E-commerce [18, 7], to name a few.”
Learning Cross-Modal Common Representations by Private–Shared Subspaces Separation (2020, methodology): “It should be noted that the original experiments in Corr-AE [17] and ACMR [7] report the MAP scores top 50 returned results, while the rest returned results are not considered.”
ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language (2019, background): “, image captioning [24, 55, 60, 34], image-text matching [13, 28, 31, 21, 14], and text-to-image generation [49, 14, 53]), only very few researchers focus on the joint domain of vision and language in 3D.”
Deep Multimodal Representation Learning: A Survey (2019, background): “[11] proposed to learn a couple of independent while correlated representation for each modality.”
A survey of deep neural network architectures and their applications (2017, methodology): “More specifically, to address the crossmodal retrieval tasks, a correspondence AE (Corr-AE) was introduced by correlating the hidden representations of two uni-modal AEs [44].”
Weakly-Shared Deep Transfer Networks for Heterogeneous-Domain Knowledge Propagation (2015, background): “As one of the variants of deep learning models, SAEs have been widely used for face recognition [15], motion detection [35], multimedia retrieval [22, 9], etc.”
Self-Supervised Adversarial Hashing Networks for Cross-Modal Retrieval (2018, methodology): “Most existing shallow cross-modal hashing methods (in both unsupervised [2, 10, 14, 18] and supervised settings [7, 17, 19, 26, 30, 40, 33]), always capture the semantic relevance in a common Hamming space.”
Adversarial Cross-Modal Retrieval (2017, methodology): “The compared methods are: 1) CCA [9], CCA-3V [7], LCFS [31], JRL [40] and JFSSL [30], which are traditional cross-modal retrieval methods, and 2) Multimodal-DBN [28], Bimodal-AE [20], Corr-AE [5], and CMDN [21], which are DNN based.”