Cross-Modal Scene Networks

By Yusuf Aytar, Lluís Castrejón, Carl Vondrick, Hamed Pirsiavash, Antonio Torralba. 2016

119 citations recorded by the source corpus.

People can recognize scenes across many different modalities beyond natural images. In this paper, we investigate how to learn cross-modal scene representations that transfer across modalities. To study this problem, we introduce a new cross-modal scene dataset. While convolutional neural networks can categorize scenes well, they also learn an intermediate representation not aligned across modalities, which is undesirable for cross-modal transfer applications. We present methods to regularize cross-modal convolutional neural networks so that they have a shared representation that is agnostic of the modality. Our experiments suggest that our scene representation can help transfer representations across modalities for retrieval. Moreover, our visualizations suggest that units emerge in the shared representation that tend to activate on consistent concepts independently of the modality.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, background): “More recently, CoupledGANs [28] and cross-modal scene networks [1] use a weight-sharing strategy to learn a common representation across domains.”
DualGAN: Unsupervised Dual Learning for Image-to-Image Translation (2017, neutral): “, [2, 9], does not lead to effective general-purpose solutions as its applicability is taskdependent, leading to unnatural image translation results, as shown in comparative studies by CycleGAN [26].”
Playing hard exploration games by watching YouTube (2018, methodology): “In this scenario, another common approach for domain alignment involves solving a shared auxiliary objective across the domains [5, 6].”
DA-GAN: Instance-Level Image Translation by Deep Attention Generative Adversarial Networks (2018, background): “CoGAN [20] and cross modal scene networks [2] use a weight-sharing strategy to learn a common representation across domains.”
Seeing Voices and Hearing Faces: Cross-Modal Biometric Matching (2018, background): “Similar to the motivation in [3], the rationale behind this architecture is to force early layers to specialise to modality specific features (such as edges in face images and spectral patterns in audio segments), while allowing later layers to capture higher-level latent cross-modal variables (such”
Image-to-image translation for cross-domain disentanglement (2018, background): “Moreover, we do not need image labels, as opposed to specialized approaches such as [4].”
PRNet: Self-Supervised Learning for Partial-to-Partial Registration (2019, methodology): “Inspired by this observation, self-supervised learning usually involves predicting part of an input from another part [47, 48], solving one task using features learned from another task [45] and/or enforcing consistency from different views/modalities [49, 50].”
Deep Multimodal Representation Learning: A Survey (2019, background): “Another advantage is that the shared common subspace tends to be modality-invariant, which is helpful for transferring knowledge from one modality to another [1], [73].”
Learning from Multiview Correlations in Open-domain Videos (2018, background): “Within a scene classification task, [11] use one neural network per modality.”
Generative Models of Visually Grounded Imagination (2017, background): “It is also related to the infamous “symbol grounding problem” in philosophy (Harnard, 1990; Baroni, 2016).”
Cross-Modal Deep Variational Hand Pose Estimation (2018, methodology): “[1] use several CNNs to co-embed data from different data modalities for scene classification and Ngiam et al.”
Recipe1M: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images (2018, background): “Previous work on cross-modality training [29], [30] suggests training models for different modalities separately and ﬁne tuning them jointly afterwards to allow alignment.”