Learning Coupled Feature Spaces for Cross-Modal Matching

By Kaiye Wang, Ran He, Liang Wang, Tien-Ping Tan. 2013

238 citations recorded by the source corpus.

Cross-modal matching has recently drawn much attention due to the widespread existence of multimodal data. It aims to match data from different modalities, and generally involves two basic problems: the measure of relevance and coupled feature selection. Most previous works mainly focus on solving the first problem. In this paper, we propose a novel coupled linear regression framework to deal with both problems. Our method learns two projection matrices to map multimodal data into a common feature space, in which cross-modal data matching can be performed. And in the learning procedure, the ell_21-norm penalties are imposed on the two projection matrices separately, which leads to select relevant and discriminative features from coupled feature spaces simultaneously. A trace norm is further imposed on the projected data as a low-rank constraint, which enhances the relevance of different modal data with connections. We also present an iterative algorithm based on half-quadratic minimization to solve the proposed regularized linear regression problem. The experimental results on two challenging cross-modal datasets demonstrate that the proposed method outperforms the state-of-the-art approaches.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces (2017, neutral): “88 LCFS [39] 41.”
Transductive Multi-View Zero-Shot Learning (2015, methodology): “Recognition could now be achieved using NN classiﬁcation with the embedded prototypes/N-shots as labelled data.”
Adversarial Cross-Modal Retrieval (2017, methodology): “The compared methods are: 1) CCA [9], CCA-3V [7], LCFS [31], JRL [40] and JFSSL [30], which are traditional cross-modal retrieval methods, and 2) Multimodal-DBN [28], Bimodal-AE [20], Corr-AE [5], and CMDN [21], which are DNN based.”
Learning Consistent Feature Representation for Cross-Modal Multimedia Retrieval (2015, methodology): “We evaluated them both with the PCA dimension reduction preprocess and without it [33], [44], where 95% information energy was preserved by the PCA.”
Cross-Modal Similarity Learning: A Low Rank Bilinear Formulation (2014, methodology): “The proposed formulation is introduced and derived to ﬁnd the solution detailed in the next part.”
Cross-Modality Submodular Dictionary Learning for Information Retrieval (2014, unknown): “Cross-modality information retrieval approaches [ 8 ] , [ 29 ] , [ 30 ] , [ 32 ] , [ 43 ] On the top of the above scenarios, we introduce a novel framework which aims at enhancing the classical unimodal information retrieval systems with unlabeled training data in a different media form.”
Learning Cross-Modal Common Representations by Private–Shared Subspaces Separation (2020, methodology): “In addition to the CCA and its extensions, some other methods that utilize advanced schemes, such as factor analysis [25], dictionary learning [26], and half-quadratic optimization [27], to learn more effective common representations.”
Dual-Path Convolutional Image-Text Embedding (2017, methodology): “In addition to the ranking loss, how can we resort to the classification loss prevalently used in the early days of cross-media analysis? To our knowledge, some early works use human annotated broad classes for cross-media retrieval on the class level [30, 35, 38], indicating the effectiveness of cl”
A Comprehensive Survey on Cross-modal Retrieval (2016, methodology): “…methods Subspace learning methods CDFE [45], GMA [15], I 2 SCA [46], PFAR [47], JRL [48], SliM 2 [49], cluster-CCA [50], CCA-3V [51], LCFS [52], JFSSL [53] Topic model SupDocNADE [54], NPBUS [55], M 3 R [56] Deep learning methods RE-DNN [57], deep-SM [58], MDNN [59] Binary representation…”
Dual-path Convolutional Image-Text Embeddings with Instance Loss (2017, background): “[12] learn the shared image-text space based on coupled input with class regression.”
Learning Compact Binary Face Descriptor for Face Recognition (2015, unknown): “8 LCFS [61] 35.”