SoundNet: Learning Sound Representations from Unlabeled Video

By Yusuf Aytar, Carl Vondrick, Antonio Torralba. 2016

1,112 citations recorded by the source corpus.

We learn rich natural sound representations by capitalizing on large amounts of unlabeled sound data collected in the wild. We leverage the natural synchronization between vision and sound to learn an acoustic representation using two-million unlabeled videos. Unlabeled video has the advantage that it can be economically acquired at massive scales, yet contains useful signals about natural sound. We propose a student-teacher training procedure which transfers discriminative visual knowledge from well established visual recognition models into the sound modality using unlabeled video as a bridge. Our sound representation yields significant performance improvements over the state-of-the-art results on standard benchmarks for acoustic scene/object classification. Visualizations suggest some high-level semantics automatically emerge in the sound network, even though it is trained without ground truth labels.

Subfields: Developmental Biology, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Developmental Biology

  1. Andrew Zisserman
  2. Lorenzo Torresani
  3. Xavier Serra
  4. Du Tran
  5. Antonio Torralba
  6. Hang Zhao
  7. Kristen Grauman
  8. Herman Kamper

In Multimodal Learning

  1. Kristen Grauman
  2. Andrew Zisserman
  3. Ruohan Gao
  4. Cordelia Schmid
  5. Arsha Nagrani
  6. David F. Harwath
  7. James R. Glass
  8. Antonio Torralba

In Representation & Self-Supervised Learning

  1. Andrew Zisserman
  2. Andrea Vedaldi
  3. Ishan Misra
  4. Kristen Grauman
  5. Cordelia Schmid
  6. Antonio Torralba
  7. James R. Glass
  8. Lorenzo Torresani

In Signal Processing

  1. Kristen Grauman
  2. James R. Glass
  3. Ruohan Gao
  4. Xavier Serra
  5. Andrew Zisserman
  6. Eduardo Fonseca
  7. David F. Harwath
  8. Lorenzo Torresani

Citations in the corpus