Self-Supervised Moving Vehicle Tracking With Stereo Sound

By Chuang Gan, Hang Zhao, Peihao Chen, David D. Cox, Antonio Torralba. 2019

159 citations recorded by the source corpus.

Humans are able to localize objects in the environment using both visual and auditory cues, integrating information from multiple modalities into a common reference frame. We introduce a system that can leverage unlabeled audiovisual data to learn to localize objects (moving vehicles) in a visual reference frame, purely using stereo sound at inference time. Since it is labor-intensive to manually annotate the correspondences between audio and object bounding boxes, we achieve this goal by using the co-occurrence of visual and audio streams in unlabeled videos as a form of self-supervision, without resorting to the collection of ground truth annotations. In particular, we propose a framework that consists of a vision ``teacher'' network and a stereo-sound ``student'' network. During training, knowledge embodied in a well-established visual vehicle detection model is transferred to the audio domain using unlabeled videos as a bridge. At test time, the stereo-sound student network can work independently to perform object localization using just stereo audio and camera meta-data, without any visual input. Experimental results on a newly collected Auditory Vehicles Tracking dataset verify that our proposed approach outperforms several baseline approaches. We also demonstrate that our cross-modal auditory localization approach can assist in the visual localization of moving vehicles under poor lighting conditions.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, background): “It was shown that KD has several benefits for optimization and knowledge transfer between tasks [59, 13, 14].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “This idea has been further developed for visually-guided audio source separation [43–48], sound generation from silent videos [39, 49], sound localization in video frames [50, 51], face and voice association [52, 53], and video-based audio localization [54, 55].”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “Gan [154] X Video frame Sound One 7 Feature soft labels 7”
Location-Aware Graph Convolutional Networks for Video Question Answering (2020, background): “Recently, deep learning has witnessed a great process (Tan, Tsang, and Wang 2014; Cao et al. 2018; 2019; Gan et al. 2019; Guo et al. 2019a).”
Listen to Look: Action Recognition by Previewing Audio (2019, background): “Other forms of cross-modal distillation consider transferring supervision from RGB to flow or depth [27] or from a visual network to an audio network, or vice versa [7, 43, 3, 18].”
RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning (2020, background): “Video analysis [24] has been a prominent research topic in computer vision due to its vast potential applications, such as action recognition [32, 8], event detection [14], action localization [4, 49, 50], audio-visual scene analysis [16, 5, 12], etc .”
Self-Supervised Learning of Audio-Visual Objects from Video (2020, background): “In a related line of work, [24] distill visual object detectors into an audio model using stereo sound, while [26] use spatial information in a scene to convert mono sound to stereo.”
Look, Listen, and Act: Towards Audio-Visual Embodied Navigation (2019, background): “network to see and hear many unlabeled videos to localize the objects that sound [24], [25], [?], [26], [27], [28], [29], [30], [31], [32].”