Audioclip: Extending Clip to Image, Text and Audio

By Andrey Guzhov, Federico Raue, Jörn Hees, Andreas R. Dengel. 2021

572 citations recorded by the source corpus.

The rapidly evolving field of sound classification has greatly benefited from the methods of other domains. Today, the trend is to fuse domain-specific tasks and approaches together, which provides the community with new outstanding models.We present AudioCLIP – an extension of the CLIP model that handles audio in addition to text and images. Utilizing the AudioSet dataset, our proposed model incorporates the ESResNeXt audio-model into the CLIP framework, thus enabling it to perform multimodal classification and keeping CLIP’s zero-shot capabilities.AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task and out-performs others by reaching accuracies of 97.15 % on ESC-50 and 90.07 % on UrbanSound8K. Further, it sets new baselines in the zero-shot ESC-task on the same datasets (69.40 % and 68.78 %, respectively).We also asses the influence of different training setups on the final performance of the proposed model. For the sake of reproducibility, our code is published.

Subfields: Music, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Music

  1. Huaming Wang
  2. Benjamin Elizalde

In Multimodal Learning

  1. Hongsheng Li
  2. Peng Gao
  3. Zhe Gan
  4. Jiebo Luo
  5. Yu Qiao
  6. Ali Farhadi
  7. Renrui Zhang
  8. Rowan Zellers

In Signal Processing

  1. Shujie Liu
  2. Yu Wu
  3. Sanyuan Chen
  4. Zhuo Chen
  5. Jinyu Li
  6. Bo Xu
  7. Yu Qiao
  8. Huaming Wang

Citations in the corpus