PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition
By Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, Mark D. Plumbley. 2019
1,568 citations recorded by the source corpus.
Audio pattern recognition is an important research topic in the machine learning area, and includes several tasks such as audio tagging, acoustic scene classification, music classification, speech emotion classification and sound event detection. Recently, neural networks have been applied to tackle audio pattern recognition problems. However, previous systems are built on specific datasets with limited durations. Recently, in computer vision and natural language processing, systems pretrained on large-scale datasets have generalized well to several tasks. However, there is limited research on pretraining systems on large-scale datasets for audio pattern recognition. In this paper, we propose pretrained audio neural networks (PANNs) trained on the large-scale AudioSet dataset. These PANNs are transferred to other audio related tasks. We investigate the performance and computational complexity of PANNs modeled by a variety of convolutional neural networks. We propose an architecture called Wavegram-Logmel-CNN using both log-mel spectrogram and waveform as input feature. Our best PANN system achieves a state-of-the-art mean average precision (mAP) of 0.439 on AudioSet tagging, outperforming the best previous system of 0.392. We transfer PANNs to six audio pattern recognition tasks, and demonstrate state-of-the-art performance in several of those tasks. We have released the source code and pretrained models of PANNs: https://github.com/qiuqiangkong/audioset_tagging_cnn.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- James R. Glass
- Yoshua Bengio
- Jianfeng Gao
- Quoc V. Le
- Furu Wei
- Trevor Darrell
- Percy Liang
In Representation & Self-Supervised Learning
- James R. Glass
- Alexei Baevski
- Yu-An Chung
- Aäron van den Oord
- Michael Auli
- Andrew Zisserman
- Wei-Ning Hsu
- Boqing Gong
In Speech & Audio
- James R. Glass
- Wei-Ning Hsu
- Hung-yi Lee
- Lei He
- Alexei Baevski
- Jinyu Li
- Yu-An Chung
- Aäron van den Oord
In Signal Processing
- James R. Glass
- Xavier Serra
- Eduardo Fonseca
- Jinyu Li
- Wei-Ning Hsu
- Hung-yi Lee
- Shinji Watanabe
- Dong Yu
Citations in the corpus
- Perceiver: General Perception with Iterative Attention (2021, methodology): “With-out these improvements the CNN-14 model does slightly worse than the Perceiver (37.5 mAP).”
- Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “For TTA, the model is evaluated standard Frechet Audio Distance (FAD) (Kilgour et al., 2019), Frechet Distance (FD) and KL divergence (KLD) based on the pre-trained audio event tagger PANN (Kong et al., 2019), and Inception score (IS) (Salimans et al., 2016).”
- FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, methodology): “FSD50K is an open and stable dataset aimed at complementing AudioSet in order to foster reproducible large-vocabulary SER research.”
- VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, methodology): “Similar to [52], we use mixup [107] on inputlabel (x-y) pairs in a mini-batch as below:”
- WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research (2023, methodology): “Second, classes in AudioSet are very unbalanced and most of the audio clips only have a single label [55], therefore, using T5 to transform tags to captions makes the training data unbalanced and noisy.”
- Towards Learning a Universal Non-Semantic Representation of Speech (2020, methodology): “Generally, fine-tuning matches or outperforms the performance of fully-supervised models trained on the downstream tasks [1, 16], especially when the amount of labeled data is small.”
- BEATs: Audio Pre-Training with Acoustic Tokenizers (2022, methodology): “In addition, to push the performance for audio classification tasks with scarce data, some previous works [Kong et al., 2020, Verbitskiy et al., 2022, Gong et al., 2021a, Chen et al., 2022a, Koutini et al., 2021, Xu et al., 2022] report the results on ESC-50 (1.6K training samples) with an…”
- VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs (2021, methodology): “As the audio backbone, we use CNN14 [21] trained on AudioSet [14] to recognize 527 acoustic events.”
- WavThruVec: Latent speech representation as intermediate features for neural speech synthesis (2022, methodology): “We collected only audio recordings in English and performed the following preprocessing steps: (1) each sample was processed using the NeMo toolkit [29] for voice activity detection to discard unspoken fragments, (2) followed by speaker diarization to ensure there is a single speaking person per cli”
- The Benefit of Temporally-Strong Labels in Audio Event Classification (2021, background): “Current state-of-the-art systems include effective techniques to mitigate the large class imbalance and benefit from data augmentation [8].”
- Improving Sound Event Classification by Increasing Shift Invariance in Convolutional Neural Networks (2021, methodology): “Convolutional Neural Networks (CNNs) have been one of the cornerstones of Sound Event Classification or Tagging (SET) in recent years [1, 2, 3, 4].”
- Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation (2022, methodology): “We select two models, PANN [10] and HTSAT [12], to construct the audio encoder.”
- Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap (2022, background): “Nevertheless, prior studies have shown that it is necessary to fine-tune several or all layers on the target task to get good downstream performance [20], [42], [43], [58].”
- Multi-Format Contrastive Learning of Audio Representations (2021, methodology): “Recent development in this direction has mainly been focusing on supervised learning [16, 17, 18, 19].”
- AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models (2023, positive): “FD, IS, and KL are based on the state-of-the-art audio classification model PANNs [25].”
- Self-Supervised Learning from Automatically Separated Sound Scenes (2021, methodology): “For example, most previous works use larger d values, ranging from 512 to 6144; we expect performance to improve to some extent as d increases [34]— the embedding representation contains more information that can be leveraged by the shallow model in the downstream task.”