SoundNet: Learning Sound Representations from Unlabeled Video
By Yusuf Aytar, Carl Vondrick, Antonio Torralba. 2016
1,112 citations recorded by the source corpus.
We learn rich natural sound representations by capitalizing on large amounts of unlabeled sound data collected in the wild. We leverage the natural synchronization between vision and sound to learn an acoustic representation using two-million unlabeled videos. Unlabeled video has the advantage that it can be economically acquired at massive scales, yet contains useful signals about natural sound. We propose a student-teacher training procedure which transfers discriminative visual knowledge from well established visual recognition models into the sound modality using unlabeled video as a bridge. Our sound representation yields significant performance improvements over the state-of-the-art results on standard benchmarks for acoustic scene/object classification. Visualizations suggest some high-level semantics automatically emerge in the sound network, even though it is trained without ground truth labels.
Subfields: Developmental Biology, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Developmental Biology
- Andrew Zisserman
- Lorenzo Torresani
- Xavier Serra
- Du Tran
- Antonio Torralba
- Hang Zhao
- Kristen Grauman
- Herman Kamper
In Multimodal Learning
- Kristen Grauman
- Andrew Zisserman
- Ruohan Gao
- Cordelia Schmid
- Arsha Nagrani
- David F. Harwath
- James R. Glass
- Antonio Torralba
In Representation & Self-Supervised Learning
- Andrew Zisserman
- Andrea Vedaldi
- Ishan Misra
- Kristen Grauman
- Cordelia Schmid
- Antonio Torralba
- James R. Glass
- Lorenzo Torresani
In Signal Processing
- Kristen Grauman
- James R. Glass
- Ruohan Gao
- Xavier Serra
- Andrew Zisserman
- Eduardo Fonseca
- David F. Harwath
- Lorenzo Torresani
Citations in the corpus
- VideoBERT: A Joint Model for Video and Language Representation Learning (2019, background): “Since most videos contain synchronized audio and visual signals, the two modalities can supervise each other to learn strong self-supervised video representations [3, 20, 21].”
- Learning Video Representations from Textual Web Supervision (2020, background): “Prior works have leveraged ambient sound [52, 5, 4, 71, 34, 51, 55], dialogue [49], and narration [70, 2, 72, 73, 46, 45, 3], all of which of which serve as useful signals.”
- Self-Supervised MultiModal Versatile Networks (2020, background): “Cross-modal teacher-student methods [10, 62] exploit the temporal co-occurrence between visual and audio modalities in a video to learn good representations.”
- Multi-task Self-Supervised Visual Learning (2017, methodology): “The motion segmentation task uses the SoundNet dataset [3].”
- Generating Videos with Scene Dynamics (2016, background): “Finally, this paper is related to a growing body of work that capitalizes on large amounts of unlabeled video for visual recognition tasks [19, 48, 38, 13, 25, 26, 3, 33, 27, 28, 20, 42, 43, 1].”
- Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, background): “Since videos contain both visual and audio signals that are synchronized, the two modalities can supervised each other [3, 18, 17, 37].”
- Look, Listen and Learn (2017, background): “In a series of recent and inspiring papers [2, 10, 20, 21], the group at MIT has investigated precisely this.”
- Time-Contrastive Networks: Self-Supervised Learning from Video (2017, background): “For example, [23, 24] used co-occurrence of sounds and visual cues in videos to learn meaningful visual features.”
- Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning (2017, neutral): “Likewise, Aytar et al (2016) introduced a method for transferring object labels from visual CNNs to audio CNNs.”
- Objects that Sound (2017, background): “There has been a recent surge of interest in cross-modal learning from images and audio [3, 4, 16, 24].”
- Contrastive Representation Distillation (2019, methodology): “Prior cross-modal work Aytar et al. (2016); Hoffman et al. (2016b;a) uses either L 2 regression or KL-divergence.”
- Moments in Time Dataset: One Million Videos for Event Understanding (2018, methodology): “[4] transfer visual models into sound for auditory analysis, and Hershey et al.”
- Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (2018, methodology): “For all tasks, both AVTS and L 3 -Net are pretrained on Kinetics, except for the evaluations on ESC-50 and DCASE2014 where Flickr-Soundnet [20] is used for pretraining.”
- Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval (2018, methodology): “We keep image encoder (e.g., pre-trained CNN) � xed in this work.”
- Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input (2018, background): “State of the art systems are trained using bounding box annotations for the training data (Girshick et al. 2013; Redmon et al. 2016), however other works investigate weakly-supervised or unsupervised object localization (Bergamo et al. 2014; Cho et al. 2015; Cinbis et al. 2016; Zhou et al. 2015).”
- AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “Some works propose proxy tasks to learn representations for downstream tasks such as classification [9, 10, 12, 13, 52–54].”
- Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, background): “Examples of commonly used multi-view data include the motion and appearance of a video [77], depth and appearance [30, 88], luminance and chrominance of an image [77, 88], or as in our work sound and video [2, 4, 12, 59].”
- Unsupervised Learning of Semantic Audio Representations (2017, background): “Recent efforts have extended this principle of self-supervision to joint audio-visual models that learn speech or audio embeddings using semantic constraints imposed by the companion visual signal [24, 25, 26, 27].”
- The Sound of Pixels (2018, methodology): “For example, [29] used audio to supervise visual representations, [30,3] used vision to supervise audio representations, and [15] used sound and vision to jointly supervise each other.”
- Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings (2019, methodology): “Originally they used the Flickr dataset [3], and subsequently a subset of the AudioSet dataset [5].”