AST: Audio Spectrogram Transformer

By Yuan Gong, Yu-An Chung, James R. Glass. 2021

1,388 citations recorded by the source corpus.

In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Yejin Choi
  3. Jianfeng Gao
  4. Luke S. Zettlemoyer
  5. Yoshua Bengio
  6. Percy Liang
  7. James R. Glass
  8. Quoc V. Le

In Model Architectures

  1. Chen Sun
  2. Anurag Arnab
  3. Alexei Baevski
  4. Cordelia Schmid
  5. Christoph Feichtenhofer
  6. Yejin Choi
  7. Andrey Guzhov
  8. Ying Shan

In Speech & Audio

  1. James R. Glass
  2. Yu-An Chung
  3. Alexei Baevski
  4. Wei-Ning Hsu
  5. Michael Auli
  6. Hung-yi Lee
  7. David F. Harwath
  8. Karen Livescu

In Signal Processing

  1. James R. Glass
  2. Yu-An Chung
  3. Alexei Baevski
  4. Wei-Ning Hsu
  5. Michael Auli
  6. Hung-yi Lee
  7. David F. Harwath
  8. Aren Jansen

Citations in the corpus