See, Hear, and Read: Deep Aligned Representations

By Yusuf Aytar, Carl Vondrick, Antonio Torralba. 2017

144 citations recorded by the source corpus.

We capitalize on large amounts of readily-available, synchronous data to learn a deep discriminative representations shared across three major natural modalities: vision, sound and language. By leveraging over a year of sound from video and millions of sentences paired with images, we jointly train a deep convolutional network for aligned representation learning. Our experiments suggest that this representation is useful for several tasks, such as cross-modal retrieval or transferring classifiers between modalities. Moreover, although our network is only trained with image+text and image+sound pairs, it can transfer between text and sound as well, a transfer the network never observed during training. Visualizations of our representation reveal many hidden units which automatically emerge to detect concepts, independent of the modality.

Subfields: Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Objects that Sound (2017, background): “In this paper our interest is in cross-modal learning from images and audio [4, 5, 16, 18, 23, 24].”
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval (2018, methodology): “4, we provide the result when a shared representation between image, text and audio modality is learned using proposed loss following the idea in [2] and used for video-text retrieval task.”
Zero-Shot Object Detection (2018, background): “Recently, [4] used images, text, and sound for generating deep discriminative representations which are shared across the three modalities.”
Multilevel Language and Vision Integration for Text-to-Clip Retrieval (2018, background): “Second, we improve upon vector embedding approaches that pool the hidden states of a recurrent neural network to obtain sentence embeddings (Hendricks et al. 2017; Gao et al. 2017; Aytar, Vondrick, and Torralba 2017).”
Learning to Separate Object Sounds by Watching Unlabeled Video (2018, methodology): “For applications in cross-modal retrieval or zero-shot classiﬁcation, other methods aim to learn aligned representations across modalities, e.g., audio, text, and visual [6].”
2.5D Visual Sound (2018, background): “Recent work explores self-supervision for visual [34, 2] and audio [4] feature learning, cross-modal representations [5], and audio-visual alignment [32, 24, 16].”
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild (2018, background): “More recent work [3, 5, 6, 53] has explored this concept by exploiting the correspondence between synchronous audio and visual data in teacher-student style architectures [5, 6], or as a form of “self-supervision” [3] where networks for both modalities are learnt from scratch (an idea that was…”
SoundSpaces: Audio-Visual Navigation in 3D Environments (2019, background): “This includes interesting directions for synthesizing sounds for video [71,16,110], spatializing sound [65,32], sound source separation [109,31,70,26,33], cross-modal feature learning [106,107,72,30], AV tracking [34,8,9,2], and learning material properties [71].”
Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision (2020, background): “Therefore, audiovisual fusion can make full use of complementary information and become an extensive tendency in computer vision and speech recognition communities [1,2,32,26,10,19].”
Self-Supervised Moving Vehicle Tracking With Stereo Sound (2019, background): “We expect the feature representations of two modalities to be close enough under certain distance metrics.”
Learning to Localize Sound Source in Visual Scenes (2018, background): “The recent years have witnessed significant progress in understanding the correlation between sound and visual signals in videos [2, 21, 20, 27, 3, 1].”
Cross Modal Audio Search and Retrieval with Joint Embeddings Based on Text and Audio (2019, background): “Similar problems are observed with images and text, and indeed there has been work to learn joint embeddings with these modalities [9, 10].”
Deep Audio-visual Learning: A Survey (2020, methodology): “Unlike other retrieval tasks such as the text-image task [48, 49, 50] or the sound-text task [51] , the audio-visual retrieval task mainly focuses on subspace learning.”
Uncertainty-Aware Audiovisual Activity Recognition Using Deep Bayesian Variational Inference (2018, background): “Multimodal models are proposed for audiovisual analysis tasks such as emotion recognition [42], audiovisual speech recognition [33], speech localization [14, 34], cross-modal retrieval [4].”