AST: Audio Spectrogram Transformer
By Yuan Gong, Yu-An Chung, James R. Glass. 2021
1,388 citations recorded by the source corpus.
In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.
Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yejin Choi
- Jianfeng Gao
- Luke S. Zettlemoyer
- Yoshua Bengio
- Percy Liang
- James R. Glass
- Quoc V. Le
In Model Architectures
- Chen Sun
- Anurag Arnab
- Alexei Baevski
- Cordelia Schmid
- Christoph Feichtenhofer
- Yejin Choi
- Andrey Guzhov
- Ying Shan
In Speech & Audio
- James R. Glass
- Yu-An Chung
- Alexei Baevski
- Wei-Ning Hsu
- Michael Auli
- Hung-yi Lee
- David F. Harwath
- Karen Livescu
In Signal Processing
- James R. Glass
- Yu-An Chung
- Alexei Baevski
- Wei-Ning Hsu
- Michael Auli
- Hung-yi Lee
- David F. Harwath
- Aren Jansen
Citations in the corpus
- MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “We note that our Audio Encoder AST is quite different from the one proposed by [46].”
- Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2021, methodology): “Some borrow the supervised learning paradigm and encoder architectures for image representation learning and replicate their success in learning audio representations (Dai et al., 2017; Boddapati et al., 2017; Kumar et al., 2018; Guzhov et al., 2021b; Gong et al., 2021).”
- Audioclip: Extending Clip to Image, Text and Audio (2021, methodology): “To successfully solve this task, different approaches were proposed that included the use of one-[27,28] or two-dimensional Convolutional Neural Networks (CNN) operating on static [18,24,32,9,15,17,33,8,30] or trainable [23,10] time-frequency transformation of raw audio.”
- Attention Bottlenecks for Multimodal Fusion (2021, methodology): “Vision Transformer (ViT) [18] (and a recent extension to audio – Audio Spectrogram Transformer (AST) [26]) adapts the Transformer architecture [61], originally designed for natural language processing, to process 2D inputs with minimal changes.”
- HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection (2022, methodology): “Recently, by introducing the transformer structure [13] into audio classification, the audio spectrogram transformer (AST) [14] further achieves the best performance through the selfattention mechanism and the pretrained model from computer vision.”
- UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition (2023, methodology): “Impressively, UniRepLKNet outperforms InternImage and the other models.”
- WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research (2023, methodology): “Inspired by the progress made in the domains of vision and language, Transformer-based models and training strategies have been successfully adapted in modeling audio signals [36], [37].”
- Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, methodology): “The spectrogram is then encoded with a pre-trained Audio Spectrogram Transformer (AST) [57], and the input embed-dings are built by concatenating the second and second-to-last layer features from the AST and applying a linear layer just as with the image ViT.”
- SSAST: Self-Supervised Audio Spectrogram Transformer (2021, methodology): “In the audio and speech domain, the recently proposed Audio Spectrogram Trans-former (AST) (Gong, Chung, and Glass 2021) and the Key-word Transformer (Berg, O’Connor, and Cruz 2021) also achieve new state-of-the-art performance on audio scene classification and keyword spotting.”
- InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, neutral): “Zero-shot Finetuning AudioCaps ClothoV1 ClothoV2 AudioCaps ClothoV1 ClothoV2 VIP-ANT [Zhao et al., 2021] 27.7 - - - VAST [Chen et al., 2024b AST [Gong et al., 2021] 95.60 BEATs [Chen et al., 2023a] 98.10 InternVideo2 s 2 98.60 Table 14: Results of Chat-centric Evaluation on Multiple Choice Video-QA…”
- The rise and potential of large language model based agents: a survey (2025, neutral): “Apart from the cascading paradigm [218, 219, 269], audio information encoding can also enhance the seamless integration of audio with other modalities of inputs [222].”
- Efficient Training of Audio Transformers with Patchout (2021, methodology): “Specaugment: We use SpecAugment [12] by masking up to 48 frequency bins and 192 time frames similar to [5].”
- A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, methodology): “AST is the first audio classification model where the convolution was not used and it is capable of capturing long-range frames context.”
- Masked Autoencoders that Listen (2022, methodology): “We report accuracy under 5-fold cross-validation with the same split used by [10].”