Objects that Sound

By Relja Arandjelovi'c, Andrew Zisserman. 2017

569 citations recorded by the source corpus.

In this paper our objectives are, first, networks that can embed audio and visual inputs into a common space that is suitable for cross-modal retrieval; and second, a network that can localize the object that sounds in an image, given the audio signal. We achieve both these objectives by training from unlabelled video using only audio-visual correspondence (AVC) as the objective function. This is a form of cross-modal self-supervision from video. To this end, we design new network architectures that can be trained using the AVC task for these functionalities: for cross-modal retrieval, and for localizing the source of a sound in an image. We make the following contributions: (i) show that audio and visual embedding can be learnt that enable both within-mode (e.g. audio-to-audio) and between-mode retrieval; (ii) explore various architectures for the AVC task, including those for the visual stream that ingest a single image, or multiple images, or a single image and multi-frame optical flow; (iii) show that the semantic object that sounds within an image can be localized (using only the sound, no motion or flow information); and (iv) give a cautionary tale in how to avoid undesirable shortcuts in the data preparation.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Signal Processing

Citations in the corpus

Learning Representations by Maximizing Mutual Information Across Views (2019, background): “Closely related work considers learning representations by predicting cross-modal correspondence [Arandjelovi´c and Zisserman, 2017, 2018].”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “Beyond video, information about camera motion [1, 32], scene geometry [67], or sound [3, 4] can all serve as natural sources of supervision.”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “Taking this idea into the self-supervised domain [8], multiple works use a pretext task of predicting whether visual and audio signals come from the same video [8, 9, 44, 58, 61, 72].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “The max-pool based approach [5, 8, 58] (Max+NCE) only optimizes over the clip-caption pair with the highest similarity score among the positive candidates.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “For video data, these pretext tasks learn representations by ordering video frames [1, 18, 34, 41, 51, 79, 83], tracking [62, 77], or using cross-modal signals like audio [2, 3, 19, 36, 60, 61].”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “Multi-modal information such as depth from a sensor [17], sound in a video [3, 4, 23, 53], sensors on an autonomous vehicle [2, 30, 79] etc.”
ActBERT: Learning Global-Local Video-Text Representations (2020, background): “The synchronized video-audio signal is also a good test-bed for cross-modal representation learning [3, 15].”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Relation to existing tasks The past few years have seen audio studied in computer vision tasks [245] for action classification [110,226], object categorization [125,234], source localization and tracking [14, 197, 212] and embodied navigation [33].”
What makes for good views for contrastive learning (2020, background): “Views are commonly different sensory signals, like photos and sounds [3], or different image channels [53] or slices in time [55], but may also be different “augmented" versions of the same data tensor [7].”
Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, background): “Contrastive learning has recently attracted a surge of interest and shown promising results in various areas including computer vision [18, 6], natu-ral language processing [11, 15], audio processing [49, 42], graph [20, 44] and multimodal data [4, 51].”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Multimodal self-supervised learning can be achieved by predicting whether a video has correspondence with an audio stream [4, 5, 68, 54], cross-modality clustering [3], and evolving losses [73].”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, background): “There has also been recent work on localizing sound sources using a network’s attention map [50,51,52].”
Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks (2019, background): “Some attempts have also been done to extend self-supervised learning to different modalities [12,13] or to audio representations only [14–17].”
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (2018, methodology): “Recently, Arandjelovic and Zisserman proposed the Audio-Visual Correspondence (AVC) – i.e., predicting if an audio-video pair is in a true correspondence – as a way to jointly learn both auditive and visual representations [21].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “and visual features [37–42]. This idea has been further developed for visually-guided audio source separation [43–48], sound generation from silent videos [39, 49], sound localization in video frames [50, 51], face and voice association [52, 53], and video-based audio localization [54, 55]. While ou”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, background): “Audio-Visual Correspondence (AVC) is a standard task [2, 3, 41, 62] used in audio-video cross-modal learning.”
The Sound of Pixels (2018, background): “In work related to our paper, [31,32] studied how to localize sounds in vision, however they do not separate multiple sounds from a mixed signal.”
EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset (2020, background): “The inclusion of visual features has aided in simplifying object classification and source separation [7], [8], [9], but”
Learning Words by Drawing Images (2019, background): “Recently proposed models that learn these correspondences can be used to perform tasks such as visually-guided audio source separation, or localizing source of a sound within an image or video [35, 44, 42, 17, 1, 18, 5].”