Efficient Training of Audio Transformers with Patchout

By Khaled Koutini, Jan Schlüter, Hamid Eghbalzadeh, Gerhard Widmer. Interspeech, 2021

410 citations recorded by the source corpus.

The great success of transformer-based models in natural language processing (NLP) has led to various attempts at adapting these architectures to other domains such as vision and audio. Recent work has shown that transformers can outperform Convolutional Neural Networks (CNNs) on vision and audio tasks. However, one of the main shortcomings of transformer models, compared to the well-established CNNs, is the computational complexity. In transformers, the compute and memory complexity is known to grow quadratically with the input length. Therefore, there has been extensive work on optimizing transformers, but often at the cost of degrading predictive performance. In this work, we propose a novel method to optimize and regularize transformers on audio spectrograms. Our proposed models achieve a new state-of-the-art performance on Audioset and can be trained on a single consumer-grade GPU. Furthermore, we propose a transformer model that outperforms CNNs in terms of both performance and training speed. Source code: https://github.com/kkoutini/PaSST

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/c397d0e17ced17e72aa3fc0df645eeabcabc32de

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yossi Adi
2. Gabriel Synnaeve
3. Furu Wei
4. Wei-Ning Hsu
5. Zhou Zhao
6. Mohit Bansal
7. Jianfeng Gao
8. Jinyu Li

In Signal Processing

1. Yossi Adi
2. Rongjie Huang
3. Jinyu Li
4. Gabriel Synnaeve
5. Shujie Liu
6. Wei-Ning Hsu
7. Zhou Zhao
8. Sanyuan Chen

Citations in the corpus

• AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, background): “The KL metrics measure the similarity between generation and target audio with the label calculated by the patch-out transformer (Koutini et al., 2021).”
• AudioGen: Textually Guided Audio Generation (2022, methodology): “Similarly to Yang et al. (2022) we additionally compute the KL-Divergence between the output of a state-of-the-art audio classification model (Koutini et al., 2021) while feeding it both the original samples and the generated audio.”
• Simple and Controllable Music Generation (2023, methodology): “[2022], we use a state-of-the-art audio classifier trained for classification on AudioSet [Koutini et al., 2021] to compute the KL-divergence over the probabilities of the labels between the original and the generated music.”
• BEATs: Audio Pre-Training with Acoustic Tokenizers (2022, methodology): “…audio classification tasks with scarce data, some previous works [Kong et al., 2020, Verbitskiy et al., 2022, Gong et al., 2021a, Chen et al., 2022a, Koutini et al., 2021, Xu et al., 2022] report the results on ESC-50 (1.6K training samples) with an additional round of supervised pre-training on…”
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, methodology): “Following common practice (Yang et al., 2022; Iashin & Rahtu, 2021), the key automated performance metrics used are melception-based (Koutini et al., 2021) FID (Heusel et al., 2017) and KL divergence to measure audio fidelity.”
• Masked Autoencoders that Listen (2022, methodology): “MBT [11] uses ImageNet-21K pre-trained ViT; AST [10] and PaSST [28] employ DeiT [14] as the Transformer backbone.”