HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection

By Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, Shlomo Dubnov. 2022

442 citations recorded by the source corpus.

Audio classification is an important task of mapping audio samples into their corresponding labels. Recently, the transformer model with self-attention mechanisms has been adopted in this field. However, existing audio transformers require large GPU memories and long training time, meanwhile relying on pretrained vision models to achieve high performance, which limits the model’s scalability in audio tasks. To combat these problems, we introduce HTS-AT: an audio transformer with a hierarchical structure to reduce the model size and training time. It is further combined with a token-semantic module to map final outputs into class featuremaps, thus enabling the model for the audio event detection (i.e. localization in time). We evaluate HTS-AT on three datasets of audio classification where it achieves new state-of-the-art (SOTA) results on AudioSet and ESC50, and equals the SOTA on Speech Command V2. It also achieves better performance in event localization than the previous CNN-based models. Moreover, HTS-AT requires only 35% model parameters and 15% training time of the previous audio transformer. These results demonstrate the high performance and high efficiency of HTS-AT.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Alexei Baevski
2. Michael Auli
3. Christoph Feichtenhofer
4. Luc Van Gool
5. Po-Yao (Bernie) Huang
6. Andrew Zisserman
7. Cordelia Schmid
8. Florian Metze

In Signal Processing

1. Alexei Baevski
2. Michael Auli
3. Tom Ko
4. Qiuqiang Kong
5. Chutong Meng
6. Wenwu Wang
7. Chengqi Zhao
8. Haohe Liu

Citations in the corpus

• WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research (2023, methodology): “networks (PANNs) [53] and a Transformer network-HTSAT [58] are considered, where both are pretrained on AudioSet with an audio tagging task.”
• Masked Autoencoders that Listen (2022, methodology): “Also, compared to HTS-AT [29] and PaSST [28], Audio-MAE is trained with audio under 16K sampling rate.”