CNN architectures for large-scale audio classification
By Shawn Hershey, Sourish Chaudhuri, D. Ellis, Jort Florent Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, D. Platt, Rif A. Saurous, Bryan Seybold and 3 more. 2016
3,011 citations recorded by the source corpus.
Convolutional Neural Networks (CNNs) have proven very effective in image classification and show promise for audio. We use various CNN architectures to classify the soundtracks of a dataset of 70M training videos (5.24 million hours) with 30,871 video-level labels. We examine fully connected Deep Neural Networks (DNNs), AlexNet [1], VGG [2], Inception [3], and ResNet [4]. We investigate varying the size of both training set and label vocabulary, finding that analogs of the CNNs used in image classification do well on our audio classification task, and larger training and label sets help up to a point. A model using embeddings from these classifiers does much better than raw features on the Audio Set [5] Acoustic Event Detection (AED) classification task.
Subfields: Computer Vision and Pattern Recognition, Music, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Andrew Zisserman
- Trevor Darrell
- Cordelia Schmid
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
In Music
- Xavier Serra
- Eduardo Fonseca
- Frederic Font
- Anurag Kumar
- Xavier Favory
- Dan Stowell
- D. Ellis
- Manoj Plakal
In Signal Processing
- Xavier Serra
- Eduardo Fonseca
- James R. Glass
- Kristen Grauman
- Andrew Zisserman
- Wei-Ning Hsu
- Xavier Favory
- Frederic Font
Citations in the corpus
- Randaugment: Practical automated data augmentation with a reduced search space (2019, background): “D perception [35], speech recognition [19] or audio recognition [18].”
- A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, methodology): “For audial information, we extract the feature map using the VGGish [39] followed by PCA for dimensionality reduction (R(128)).”
- SoundNet: Learning Sound Representations from Unlabeled Video (2016, background): “Concurrent work [11] also explores video as a form of weak labeling for audio event classification.”
- MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (2017, methodology): “We seeded our model architecture with a residual network based on a structure previously used for this dataset [12].”
- Measuring Catastrophic Forgetting in Neural Networks (2017, methodology): “These features were extracted using a variant ResNet-50 for audio data (Hershey et al. 2017), which was pre-trained on an early version of the YouTube-8m dataset (Abu-El-Haija et al. 2016).”
- VoxCeleb: A Large-Scale Speaker Identification Dataset (2017, background): “This has led to a very recent shift from hand-crafted features to the domain of deep CNNs which can be applied to higher dimensional inputs [13, 14] and for speaker iden-tification [15].”
- Real-Time Speech Frequency Bandwidth Extension (2020, methodology): “• VGG distance, the L2-distance between the ground truth and the reconstructed audio embeddings computed by a pre-trained VGGish network [24].”
- FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, methodology): “2) Network Architectures: Current trends in SER encompass mainly CNNs [87, 104, 112]–[114] and CRNNs [20, 115].”
- Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, methodology): “The audio features are extracted using the audio CNN [55].”
- Cross-Task Weakly Supervised Learning From Instructional Videos (2019, methodology): “Features: We represent each video segment xi using RGB I3D features [8] (1024D), Resnet-152 features [16] (2048D) extracted at each frame and averaged over one-second temporal windows, and audio features from [17] (128D).”
- MusicLM: Generating Music From Text (2023, methodology): “We report the FAD based on two audio embedding models, both of which are publicly available: (1) Trill 2 (Shor et al., 2020), which is trained on speech data, and (2) VGGish 3 , (Hershey et al., 2017) which is trained on the YouTube-8M audio event dataset (Abu-El-Haija et al., 2016).”
- Multi-modal Transformer for Video Retrieval (2020, methodology): “Audio features are extracted using VGGish model [6] trained on YT8M.”
- Use What You Have: Video retrieval using representations from collaborative experts (2019, methodology): “Audio embeddings are obtained with a VGGish model, trained for audio classification on the YouTube-8m dataset [23].”
- Unsupervised Learning of Semantic Audio Representations (2017, methodology): “The last few years have seen great advances in nonspeech audio processing, as popular deep learning architectures developed in the speech and image processing communities have been ported to this relatively understudied domain [1, 2, 3, 4].”
- PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition (2019, methodology): “However, systems trained on large-scale audio datasets have been more limited [1], [13]–[15].”
- Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification (2017, methodology): “We use ConvNet-based audio classification system [6] to extract acoustic feature.”
- Learnable pooling with Context Gating for video classification (2017, methodology): “The audio features are extracted from a CNN architecture trained for audio classification [49].”