Learning Cross-Modal Common Representations by Private–Shared Subspaces Separation

By Xing Xu, Kaiyi Lin, Lianli Gao, Heng Tao Shen, Xuelong Li. 2020

51 citations recorded by the source corpus.

Due to the inconsistent distributions and representations of different modalities (e.g., images and texts), it is very challenging to correlate such heterogeneous data. A standard solution is to construct one common subspace, where the common representations of different modalities are generated to bridge the heterogeneity gap. Existing methods based on common representation learning mostly adopt a less effective two-stage paradigm: first, generating separate representations for each modality by exploiting the modality-specific properties as the complementary information, and then capturing the cross-modal correlation in the separate representations for common representation learning. Moreover, these methods usually neglect that there may exist interference in the modality-specific properties, that is, the unrelated objects and background regions in images or the noisy words and incorrect sentences in the text. In this article, we hypothesize that explicitly modeling the interference within each modality can improve the quality of common representation learning. To this end, we propose a novel model private–shared subspaces separation (P3S) to explicitly learn different representations that are partitioned into two kinds of subspaces: 1) the common representations that capture the cross-modal correlation in a shared subspace and 2) the private representations that model the interference within each modality in two private subspaces. By employing the orthogonality constraints between the shared subspace and the private subspaces during the one-stage joint learning procedure, our model is able to learn more effective common representations for different modalities in the shared subspace by fully excluding the interference within each modality. Extensive experiments conducted on cross-modal retrieval verify the advantages of our P3S method compared with 15 state-of-the-art methods on four widely used cross-modal datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Shihao Bai
3. Aishan Liu
4. Yuqing Ma
5. Xianglong Liu
6. Jing Zhang
7. Charu C. Aggarwal
8. Jun Yu

In Computer Vision and Pattern Recognition

1. Xianglong Liu
2. Dacheng Tao
3. Shihao Bai
4. Aishan Liu
5. Edwin R. Hancock
6. Yuqing Ma
7. Ling Shao
8. Luc Van Gool

In Multimodal Learning

1. Dacheng Tao
2. Xianglong Liu

In Representation & Self-Supervised Learning

1. Dacheng Tao
2. Xianglong Liu

Citations in the corpus

• Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, background): “Notable successes include image classification [27]–[29], image generation [30]–[32], representation learning [33], [34], image retrieval [35], [36], object detection [37]–[39], video applications [40], and image translation [41].”